When to fetch records from the backend service? #37

New issue

Open

opened 2025-04-30 20:40:37 +00:00 by jsheunis · 3 comments

jsheunis commented

2025-04-30 20:40:37 +00:00

Owner

This issue relates to several other open issues:

Background of the related functionality can be found here:

When to fetch records from the backend service?

The main question is: when exactly, during the process of a user interacting with the application, should requests to the server be made to fetch records of specific types.

And a follow-up question: when should calls to the same endpoints as were made previously, be made again (or ignored).

How we approach these questions will have an important influence on the UX, both in terms of which data will be available at which points in time (meaning that some data might be unavailable when they are needed, or vice versa), and in terms of how the user experiences delay. The (un)availability of data is likely the cause of #11, while the delay is evident in #31.

How is it done currently?

At the moment, data will be fetched:

At application startup, if configured to do so via the service_fetch_before option:
- this option can take an array for get-record, with elements being specific record pids; these specific records will be fetched up front
- this option can take an array for get-records, with elements being specific class names; all records of these types will be fetched up front
When selecting a data type, e.g. Person:
- all records of the selected class will be fetched when selected
In the InstancesSelectEditor, when the component mounts:
- firstly, the component determines the specific class and all of its subclasses that have to be loaded
- then, for all of these classes, all records are fetched

In addition, multiple calls to the same endpoint are prevented by default.

How should it be done?

I am honestly not sure. I have no previous experience building such UX workflows that have to fetch data from a server constantly, so I'm curious if there might be some design patterns that I'm unaware of. The important thing is to make the UI/UX experience as seamless and as intuitive as possible, i.e. remove delays as much as possible, and render data in the most intuitive form as early in the workflow as possible.

I have a couple of ideas that would be good to get feedback on:

towards a fix for #31, should we move the data fetching step to the point of interaction? So in the InstancesSelectEditor, rather than fetching all records of the main class and all of its subclasses when the component mounts, we can fetch it when the user actually interacts with the component, i.e. when they click on the component to see the options? One could show a busy indicator inside the opened dropdown while the fetching happens.
also related to the InstancesSelectEditor, this issue has been proposed before: https://github.com/psychoinformatics-de/shacl-vue/issues/69
towards a fix for the problems related to the order in which records are fetched, we could:
- add more classes to the service_fetch_before configuration option, i.e. shift the delay to a once-off up front cost, and get the seamless and correct rendering of related records as benefit
- introduce another point into the workflow for when records should be fetched: when displaying them. This would happen after the user selects a specific type and all records of that type have been fetched. Before displaying all records of that class, the application would first run through all possible properties of that class and inspect the so-called range of those properties to see which other classes are related, and hence which types of records should be fetched in addition. This could also be done recursively, up to a prespecified level of recursion.

This issue relates to several other open issues: - https://hub.psychoinformatics.de/inm7/annotate.inm7.de-simplesubmit/issues/11 - https://hub.psychoinformatics.de/inm7/annotate.inm7.de-simplesubmit/issues/29 - https://hub.psychoinformatics.de/inm7/annotate.inm7.de-simplesubmit/issues/31 Background of the related functionality can be found here: - https://github.com/psychoinformatics-de/shacl-vue/issues/68 - https://github.com/psychoinformatics-de/shacl-vue/issues/69 ### When to fetch records from the backend service? The main question is: when exactly, during the process of a user interacting with the application, should requests to the server be made to fetch records of specific types. And a follow-up question: when should calls to the same endpoints as were made previously, be made again (or ignored). How we approach these questions will have an important influence on the UX, both in terms of which data will be available at which points in time (meaning that some data might be unavailable when they are needed, or vice versa), and in terms of how the user experiences delay. The (un)availability of data is likely the cause of #11, while the delay is evident in #31. ### How is it done currently? At the moment, data will be fetched: 1. At application startup, if configured to do so via the `service_fetch_before` option: - this option can take an array for `get-record`, with elements being specific record `pid`s; these specific records will be fetched up front - this option can take an array for `get-records`, with elements being specific class names; all records of these types will be fetched up front 2. When selecting a data type, e.g. `Person`: - all records of the selected class will be fetched when selected 3. In the `InstancesSelectEditor`, when the component mounts: - firstly, the component determines the specific class and all of its subclasses that have to be loaded - then, for all of these classes, all records are fetched In addition, **multiple calls to the same endpoint are prevented by default**. ### How should it be done? I am honestly not sure. I have no previous experience building such UX workflows that have to fetch data from a server constantly, so I'm curious if there might be some design patterns that I'm unaware of. The important thing is to make the UI/UX experience as seamless and as intuitive as possible, i.e. remove delays as much as possible, and render data in the most intuitive form as early in the workflow as possible. I have a couple of ideas that would be good to get feedback on: - towards a fix for #31, should we move the data fetching step to the point of interaction? So in the `InstancesSelectEditor`, rather than fetching all records of the main class and all of its subclasses when the component mounts, we can fetch it when the user actually interacts with the component, i.e. when they click on the component to see the options? One could show a busy indicator inside the opened dropdown while the fetching happens. - also related to the `InstancesSelectEditor`, this issue has been proposed before: https://github.com/psychoinformatics-de/shacl-vue/issues/69 - towards a fix for the problems related to the order in which records are fetched, we could: - add more classes to the `service_fetch_before` configuration option, i.e. shift the delay to a once-off up front cost, and get the seamless and correct rendering of related records as benefit - introduce another point into the workflow for when records should be fetched: when displaying them. This would happen after the user selects a specific type and all records of that type have been fetched. Before displaying all records of that class, the application would first run through all possible properties of that class and inspect the so-called range of those properties to see which other classes are related, and hence which types of records should be fetched in addition. This could also be done recursively, up to a prespecified level of recursion.

mih commented

2025-05-01 13:04:17 +00:00

Owner

Three thoughts come to my mind:

Prefetching will become progressively slower with more data being in the system. Small tasks will become slow, because prefetching cannot know what is needed. I am very much in favor of fetching close to the point where the demand is known.
This could be a gradual thing, however. When a selector drop-down is opened, this is the latest point in time to fetch items. But when a form is opened, and the app is idle, it could already start fetching types known to be needed by the visible selectors.
It is worth tracking when which types have been fetched (last). If I open a bunch of forms for the same type, the same related type-instance would need fetching, so once is enough -- unless the app is open for an hour, and in the meantime a collaborator added records.

Three thoughts come to my mind: 1. Prefetching will become progressively slower with more data being in the system. Small tasks will become slow, because prefetching cannot know what is needed. I am very much in favor of fetching close to the point where the demand is known. 2. This could be a gradual thing, however. When a selector drop-down is opened, this is the latest point in time to fetch items. But when a form is opened, and the app is idle, it could already start fetching types known to be needed by the visible selectors. 3. It is worth tracking when which types have been fetched (last). If I open a bunch of forms for the same type, the same related type-instance would need fetching, so once is enough -- unless the app is open for an hour, and in the meantime a collaborator added records.

jsheunis commented

2025-05-06 20:28:05 +00:00

Author

Owner

@mih thanks! Sorry I missed the notification for you comment. I share pretty much the same opinions as you gave. One point regarding:

This could be a gradual thing, however. When a selector drop-down is opened, this is the latest point in time to fetch items. But when a form is opened, and the app is idle, it could already start fetching types known to be needed by the visible selectors.

This is indeed how it is done currently. But the problem with the current approach is that the default behavior of a VueJS component is to mount only once all of its child components are mounted, so e.g. a dropdown component will render once its (currently known) list items are mounted. I have a strong suspicion that this leads to much of the lag that we have seen and reported in issues.

@mih thanks! Sorry I missed the notification for you comment. I share pretty much the same opinions as you gave. One point regarding: > This could be a gradual thing, however. When a selector drop-down is opened, this is the latest point in time to fetch items. But when a form is opened, and the app is idle, it could already start fetching types known to be needed by the visible selectors. This is indeed how it is done currently. But the problem with the current approach is that the default behavior of a VueJS component is to mount only once all of its child components are mounted, so e.g. a dropdown component will render once its (currently known) list items are mounted. I have a strong suspicion that this leads to much of the lag that we have seen and reported in issues.

jsheunis commented

2025-05-19 10:01:02 +00:00

Author

Owner

After some time consuming backs and forths on this behavior, the latest state can now be described as follows:

`shacl-vue` performance improvements...

Updates can be seen in the current test deployment https://annotate.inm7.de/s/test and compared to the production deployment https://annotate.inm7.de/s/users.

I made major-ish changes to the InstancesSelectEditor component (the one where you can select e.g. a User from a list of available Users) in an attempt to reduce lag and improve performance (see detailed commit message), including:

changing the underlying components to enable using vue-virtual-scroller, which speeds up rendering of long lists
not doing an up-front fetch anymore of all records that are to be displayed in the dropdown, rather fetching them only if the user clicks on the dropdown (this should fix the issue where some of these components in a just-opened form took ages to actually show up)
if the dropdown has a selected value when a form is opened, only the single record associated with this value is fetched

I also updated the NodeShapeViewer component, which displays a record in the list of records when a user selects a type, on the main page. See commit message. Mainly a few performance improvements under the hood, and then hiding some details like Annotations by default, so that there's less to render.

After some time consuming backs and forths on this behavior, the latest state can now be described as follows: ### `shacl-vue` performance improvements... Updates can be seen in the current test deployment https://annotate.inm7.de/s/test and compared to the production deployment https://annotate.inm7.de/s/users. I made major-ish changes to the `InstancesSelectEditor` component (the one where you can select e.g. a `User` from a list of available `User`s) in an attempt to reduce lag and improve performance (see [detailed commit message](https://github.com/psychoinformatics-de/shacl-vue/commit/99452af93ab42e9244ebc0f36077d136a1cbc3cf)), including: - changing the underlying components to enable using `vue-virtual-scroller`, which speeds up rendering of long lists - not doing an up-front fetch anymore of all records that are to be displayed in the dropdown, rather fetching them only if the user clicks on the dropdown (this should fix the issue where some of these components in a just-opened form took ages to actually show up) - if the dropdown has a selected value when a form is opened, only the single record associated with this value is fetched I also updated the `NodeShapeViewer` component, which displays a record in the list of records when a user selects a type, on the main page. See [commit message](https://github.com/psychoinformatics-de/shacl-vue/commit/e1a051f0a4a2c26d8b834965b785eb41ab745a0a). Mainly a few performance improvements under the hood, and then hiding some details like `Annotation`s by default, so that there's less to render.