YouEDC Capture

Scheduled data delivery and delta export

The weekly file for the statistician, the DSMB or the medical monitor on a schedule, on the portal or to SFTP, with its SHA-256; or only what changed since the previous delivery, with the reasons.

Every week the statistician, the DSMB or the medical monitor waits for the study's data. The Data deliveries page does this transfer for you: a plan says what goes to whom, in which format, when and how; each run freezes a data cut, builds the file and keeps it with its SHA-256 checksum. A plan can send the whole study each time or only what changed since the previous delivery (a delta), with the reason of each change.

A delivery plan: Run now and Edit buttons, the schedule, the recipients on the portal, the keep period; the runs of the plan with the file name, its size, the data cut, the SHA-256, the rows added, changed and removed, Download and .sha256 — A weekly delta plan to the statistician in the demo study: the first run is the full baseline, the next ones carry the changes.

Make a plan

Open the study, Data deliveries, New plan (the data manager and the project manager can; a member limited to some sites cannot, because a delivery holds every site).

Name and Recipient: for example “Weekly data for the statistician”, “Statistician of Pharma Co”.
Content: de-identified (the default: pseudonyms S0001…, each patient's dates moved by the same days, the age instead of the birth year, no free text; the same rules as the de-identified data cut) or as it is.
Full or changes: the whole study each time, or only the changes since the previous delivery.
Format: Excel, CSV, SPSS, Stata, SAS XPT or R (the formats of the export builder). SDTM is not offered here yet: it is made from the study's files in YouEDC Studio.
When: only by the “Run now” button, daily, weekly on a weekday or monthly on a day (1–28), at the hour you choose (the installation's time). Choose how many days the files are kept (90 by default).

The plan's form: name, recipient, content, full or changes, format; the schedule, weekday, day and hour, the keep period; the members who may receive it on the portal, and the SFTP server, port, user, folder, sign-in, password or private key — The plan's form: what and to whom, when, and how.

Every change of a plan is in the study's audit trail with the old and the new value; the SFTP password is never written there.

On the portal or to an SFTP server

On the portal: tick the members of the team who receive it. Only those with the right to the content are listed: the statistician receives de-identified data only, data as it is goes only to the data manager and the project manager. They get a short letter “a new delivery is ready” with a link; the file is never attached, the download needs signing in. The statistician sees on the page only the plans they receive.
To an SFTP server: give the server, the port, the user, the folder and the password or a private key (OpenSSH or PEM, without a passphrase). The secret is kept encrypted and never shown again. The file goes under a temporary name and is renamed when complete; next to it comes “<file>.sha256”.
Data as it is to SFTP needs a second confirmation: name the agreement that allows identifiable data of the study on the recipient's server (number and date) and tick the box. It is asked on every save of the plan; without it no file as it is goes to SFTP. De-identified data needs no confirmation.
After saving press Check connection: the page shows the server's host key (SHA256:…). Compare it with the one the recipient's administrator gave you, by phone or in a signed letter, tick the confirmation and press Pin this key. Until a key is pinned nothing is uploaded; a server that later presents another key is refused. A new server address needs a new pin.
For safety the portal sends only to servers on the internet: internal addresses (loopback, private, link-local) are refused unless the administrator of the installation sets DELIVERY_SFTP_PRIVATE=1 in .env.
A failed upload is tried again several times in the following minutes; when every attempt fails, the run says why and the plan's author gets a letter.

Runs and the changes file

Run now queues a run; the schedule queues its own at the hour. One run of a plan at a time. Each run freezes a new data cut named “Delivery «plan» #n”, so the file can be made again from it at any time and checked. The cuts of deliveries show on the cuts page marked “Data delivery, run n”; they are hidden by default, so they do not crowd out the cuts of interim analyses, and a button above the list shows them.

A plan of changes: the first run is the full baseline and says so in “About the file”. Each next file has:

every form sheet with only the rows added or changed since the previous delivery, whole rows in the same layout as a full file: replace the rows you have by SUBJECT, VISIT, INSTANCE (and ROW in a table's sheet);
the Changes sheet: one line per changed variable with CHANGE (added, changed, removed), the OLD and NEW value, CHANGED_AT and REASON, the reason the site gave in the audit trail;
the Removed rows sheet: rows that were in the previous delivery and are gone, with why when it is known (for example the patient withdrew the use of their data);
the patients and the dictionary in full, and “About the file”: the plan, the run, the cut and its SHA-256, the cut it is compared with, how many rows were added, changed and removed.

A de-identified delta is computed on the de-identified copies of both cuts (the pseudonym and the date shift of a patient stay the same), and it has no REASON: the reasons are free text. In SPSS, Stata, SAS and R the changes and the removed rows come as their own tables.

Checking the file, keeping and the trail

The SHA-256 of each file is on the page, in the letter and in the “.sha256” file: the recipient checks it with sha256sum -c.
A delivered file never changes. After the keep period the file is deleted; the run, its SHA-256 and its data cut stay, and the cut can be downloaded again on the cuts page.
Every run, letter, upload, download and deletion is in the study's audit trail and the export history; downloads count against the daily export limit.
The QA and training environments, demo studies, studies in the trash and studies not launched yet never send by schedule or SFTP; “Run now” on the portal still works. In a demo study the runs were written by the demo story.

Personal data and the law

De-identified data lowers the risk of recognising a patient, but under Russian law (152-FZ) it is still personal data: send it only under an agreement with the recipient that covers it. Sending data to a server outside Russia is a cross-border transfer with its own rules: check them with your organisation before setting up such a plan. Send data as it is only when the protocol and the agreements allow it.