New approach to generating/maintaining list of records + `ShaclVueStarter` feature #349

Merged

jsheunis merged 2 commits from shaclvue-kickstarter into main

2026-05-13 21:54:18 +00:00

jsheunis commented

2026-05-12 22:48:44 +00:00

Owner

New approach to generating/maintaining list of records

The premise of this change is that the getInstanceItems approach to generating the list of displayed records (from graph data) results in bad performance. This is mainly because getInstanceItems is called many times (mainly triggered by a watcher reacting to any graph changes), each time regenerating the full list of records. The proposed new approach is based on maintaining a global list of items that is incrementally updated when new nodes are added to or updated in the graph.

parseTTLandDedup is the ReactiveRdfDataset class method that receives incoming TTL strings parses it into quads, deduplicates blank nodes, and adds it to the N3 store. It already provided an array with all added quads as the return value. A new utility function, getUniqueRootNodes, is now used to determine the unique named nodes that are included in the added quads. This allows saying: 'these are the records that were just added or updated' in the graph store. parseTTLandDedup now returns both the added quads and the array of unique added/updated record IRIs, and downstream functionality using it has been updated to access its outputs correctly. A new ReactiveRdfDataset class method emitAddedRecords(records) is also added, which can be called with an array of named node (i.e. record) IRIs and which uses dispatchEvent to emit the new recordsChanged event, for which listeners can be added. emitAddedRecords is now used in multiple places in order to emit records that have just been added/updated (because of this, the previous calls to rdfDS.triggerReactivity become redundant):

In the useData composable, inside fetchRdfData right after parseTTLandDedup returns the list of unique added/updated records. This happens downstream for every call to fetchFromService, which is the standard way in which shacl-vue gets backend data in response to user interaction.
In the useWizard composable: here we only call emitAddedRecords if the context is not _record because it means there is no open form (which would need to process added quads in its own way); it means the wizard immediately adds all quads to the graph.
In the FormEditor component when the user saved the record, i.e. in the saveForm function. Here it is called specifically when the record is a PID record (i.e. not a blank node).

Now, onto the algorithmic change for getInstanceItems, which is done in the useRecords composable. As mentioned, we now maintain a global list of items (recordItemsAll and recordItemsByClass) that is incrementally updated when new nodes are added to or updated in the graph. This composable handles the new process as follows:

adds an event listener to rdfDS for the new recordsChanged event, which calls the new enqueueChanges function
enqueueChanges adds all added/updated records to a queue, and runs the code to process the queue
processQueue will keep on running as long as there are records in the queue, and will process records in batches of 10 by calling the updateRecordItem function per record
updateRecordItem runs all the code that used to be run per item by getInstanceItems, and then adds the processed record to the recordItemsAll and recordItemsByClass reactive variables
any further computations on the global lists, such as filtering based on user-typed text, are done via computed refs that depend on the above reactive refs. Important new computed refs include filteredRecordItemsAll and filteredRecordItemsByClass which filters all records and record per class based on search text, as well as filteredRecordItemsForClassWithSubclassItems which filters records per class while also including all records of a class's subclasses, which is required for the existing priority_classes/include_subclasses config feature.

The main ShaclVue component has been updated to use the adjusted useRecords composable correctly, specifically passing the new filtered computed refs to the ShaclVueRecords component, which has updated props filteredRecords and fetchedItemCount to handle the new composable structure.

Also, as part of cleanup several previous computed refs were moved as functions to the useShapes composable:

getIdFilteredNodeShapeNames
getNoEditClassList
getFilteredNodeShapeNames
getPriorityFilteredNodeShapeNames
getOrderedNodeShapeNames
getAllClassItems
These are now called on app startup during config processing.

Lastly, the useWizard composable has been updated to include a new feature needed by the upcoming kickstarter component. This feature allows a wizard to be shown not specifically for a record, nor specifically for a class, but for all classes. It's goal is to allow the configuration of very generic wizards that would be applicable for all classes. The config spec for it's selection is as follows (with AddRecordWizard being an example wizard name):

wizard_editor_selection:
  _classes:
    - AddRecordWizard

To allow distinction of the applicable class IRI when the associated wizard template is processed, the class IRI is now made part of the wizard data by default in handleWizardSave (accessible as class_uri). Also showWizardGroup and setupWizards functions are updated to handle the new wizard_editor_selection key.

Introduce new `ShaclVueStarter` app variant

This work is in response to the identified need for kick-starting the user-based entry of domain-specific knowledge into a shacl-vue metadata system, so that a critical mass for continued use of the system is available soon after deploying it for the first time. The new ShaclVueStarter component is designed to encourage users to enter minimal records in a quick, intuitive, and interactive way. It has the following features:

It serves as a new variant of the main entrypoint of a shacl-vue deployment. This requires the specification of a Vite-recognizable environment variable, VITE_SHACLVUE_VARIANT that influences whether the application is built with the (default) ShaclVue component or the new ShaclVueStarter component. An example command for building the starter app:

VITE_SHACLVUE_VARIANT=starter npm run build:app

If the environment variable is not specified, the default application will be built.

Instead of making only a single class selectable at a time, ShaclVueStarter shows all record types (aka classes), and many records per class. The user can scroll down the main view through all classes, and relevant records can be scrolled within each class "card". The left-hand-side pane now shows the list of classes with added checkboxes, which allows including/exluding classes from being displayed in the main view.
A ShaclVueStarter deployment uses the new _classes option of the wizard_selection configuration to specify a wizard that will be shown for all classes. This allows the use of a generic "add new record" wizard that only asks for a display name and description, with example config:

  AddRecordWizard:
    name: Add New Record Wizard
    tooltip: Add a new record
    icon: mdi-plus-circle-outline
    description: Add the display name and description and then hit *Save*
    inputs:
      - prop: name
        name: Display name
        description: The display name of the new record
        type: text
        placeholder: 'My record title'
        required: true
      - prop: description
        name: Description
        description: The description of the new record
        type: text-paragraph
        placeholder: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod...'
    template: content:add-record

Text search functionality provides the same UX as before, however the search now happens across all records, i.e. not only on a single class-level.
Unnecessary detail and functionality from the default ShaclVue application are not available in the starter application. This mainly means the record viewer is minimalistic, containing just the display name and a mini menu button with options to edit the record (which will open the established form) and view the record RDF.

New/updated functionality to support all of the above:

New ShaclVueStarter, ShaclVueRecordsMini, and NodeShapeViewerMini components
App.vue now dynamically loads the main entrypoint components and uses <component :is=""> to render the component that is selected based on the VITE_SHACLVUE_VARIANT value
ShaclVueStarter uses a new getFirstPages() function during app startup in order to fetch the first page from the dumpthings backend for all classes. This allows getting upfront info about number of records, and some actual records, to display on the main view that shows all classes and their records.
In ShaclVueStarter, the selectType is only called once on app startup, using the Thing class as the selected type.
ShaclVueStarter has a new computed variable, itemsByClass, which takes all filtered records as input and groups them by class. This reactive ref is provided as the input of records to ShaclVueRecordsMini per class.
As with ShaclVueRecords, ShaclVueRecordsMini emits an event when the user scrolls to the bottom of the list of records. However, ShaclVueStarter has a new function that listens for this event and will then fetch the next page of records for that specific class.
NodeShapeViewerMini has the simplistic rendering of a record display name and menu options.

## New approach to generating/maintaining list of records The premise of this change is that the `getInstanceItems` approach to generating the list of displayed records (from graph data) results in bad performance. This is mainly because `getInstanceItems` is called many times (mainly triggered by a watcher reacting to *any* graph changes), each time regenerating the full list of records. The proposed new approach is based on maintaining a global list of items that is incrementally updated when new nodes are added to or updated in the graph. `parseTTLandDedup` is the `ReactiveRdfDataset` class method that receives incoming TTL strings parses it into quads, deduplicates blank nodes, and adds it to the `N3` store. It already provided an array with all added quads as the return value. A new utility function, `getUniqueRootNodes`, is now used to determine the unique named nodes that are included in the added quads. This allows saying: 'these are the records that were just added or updated' in the graph store. `parseTTLandDedup` now returns both the added quads and the array of unique added/updated record IRIs, and downstream functionality using it has been updated to access its outputs correctly. A new `ReactiveRdfDataset` class method `emitAddedRecords(records)` is also added, which can be called with an array of named node (i.e. record) IRIs and which uses `dispatchEvent` to emit the new `recordsChanged` event, for which listeners can be added. `emitAddedRecords` is now used in multiple places in order to emit records that have just been added/updated (because of this, the previous calls to `rdfDS.triggerReactivity` become redundant): - In the `useData` composable, inside `fetchRdfData` right after `parseTTLandDedup` returns the list of unique added/updated records. This happens downstream for every call to `fetchFromService`, which is the standard way in which `shacl-vue` gets backend data in response to user interaction. - In the `useWizard` composable: here we only call `emitAddedRecords` if the context is not `_record` because it means there is no open form (which would need to process added quads in its own way); it means the wizard immediately adds all quads to the graph. - In the `FormEditor` component when the user saved the record, i.e. in the `saveForm` function. Here it is called specifically when the record is a PID record (i.e. not a blank node). Now, onto the algorithmic change for `getInstanceItems`, which is done in the `useRecords` composable. As mentioned, we now maintain a global list of items (`recordItemsAll` and `recordItemsByClass`) that is incrementally updated when new nodes are added to or updated in the graph. This composable handles the new process as follows: - adds an event listener to `rdfDS` for the new `recordsChanged` event, which calls the new `enqueueChanges` function - `enqueueChanges` adds all added/updated records to a queue, and runs the code to process the queue - `processQueue` will keep on running as long as there are records in the queue, and will process records in batches of 10 by calling the `updateRecordItem` function per record - `updateRecordItem` runs all the code that used to be run per item by `getInstanceItems`, and then adds the processed record to the `recordItemsAll` and `recordItemsByClass` reactive variables - any further computations on the global lists, such as filtering based on user-typed text, are done via computed refs that depend on the above reactive refs. Important new computed refs include `filteredRecordItemsAll` and `filteredRecordItemsByClass` which filters all records and record per class based on search text, as well as `filteredRecordItemsForClassWithSubclassItems` which filters records per class while also including all records of a class's subclasses, which is required for the existing `priority_classes`/`include_subclasses` config feature. The main `ShaclVue` component has been updated to use the adjusted `useRecords` composable correctly, specifically passing the new filtered computed refs to the `ShaclVueRecords` component, which has updated props `filteredRecords` and `fetchedItemCount` to handle the new composable structure. Also, as part of cleanup several previous computed refs were moved as functions to the `useShapes` composable: - `getIdFilteredNodeShapeNames` - `getNoEditClassList` - `getFilteredNodeShapeNames` - `getPriorityFilteredNodeShapeNames` - `getOrderedNodeShapeNames` - `getAllClassItems` These are now called on app startup during config processing. Lastly, the `useWizard` composable has been updated to include a new feature needed by the upcoming kickstarter component. This feature allows a wizard to be shown not specifically for a record, nor specifically for a class, but for all classes. It's goal is to allow the configuration of very generic wizards that would be applicable for all classes. The config spec for it's selection is as follows (with `AddRecordWizard` being an example wizard name): ```yaml wizard_editor_selection: _classes: - AddRecordWizard ``` To allow distinction of the applicable class IRI when the associated wizard template is processed, the class IRI is now made part of the wizard data by default in `handleWizardSave` (accessible as `class_uri`). Also `showWizardGroup` and `setupWizards` functions are updated to handle the new `wizard_editor_selection` key. ## Introduce new `ShaclVueStarter` app variant This work is in response to the identified need for kick-starting the user-based entry of domain-specific knowledge into a `shacl-vue` metadata system, so that a critical mass for continued use of the system is available soon after deploying it for the first time. The new `ShaclVueStarter` component is designed to encourage users to enter minimal records in a quick, intuitive, and interactive way. It has the following features: 1. It serves as a new variant of the main entrypoint of a `shacl-vue` deployment. This requires the specification of a Vite-recognizable environment variable, `VITE_SHACLVUE_VARIANT` that influences whether the application is built with the (default) `ShaclVue` component or the new `ShaclVueStarter` component. An example command for building the starter app: ```bash VITE_SHACLVUE_VARIANT=starter npm run build:app ``` If the environment variable is not specified, the default application will be built. 2. Instead of making only a single class selectable at a time, `ShaclVueStarter` shows all record types (aka classes), and many records per class. The user can scroll down the main view through all classes, and relevant records can be scrolled within each class "card". The left-hand-side pane now shows the list of classes with added checkboxes, which allows including/exluding classes from being displayed in the main view. 3. A `ShaclVueStarter` deployment uses the new `_classes` option of the `wizard_selection` configuration to specify a wizard that will be shown for all classes. This allows the use of a generic "add new record" wizard that only asks for a display name and description, with example config: ```yaml AddRecordWizard: name: Add New Record Wizard tooltip: Add a new record icon: mdi-plus-circle-outline description: Add the display name and description and then hit *Save* inputs: - prop: name name: Display name description: The display name of the new record type: text placeholder: 'My record title' required: true - prop: description name: Description description: The description of the new record type: text-paragraph placeholder: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod...' template: content:add-record ``` 4. Text search functionality provides the same UX as before, however the search now happens across all records, i.e. not only on a single class-level. 5. Unnecessary detail and functionality from the default `ShaclVue` application are not available in the starter application. This mainly means the record viewer is minimalistic, containing just the display name and a mini menu button with options to edit the record (which will open the established form) and view the record RDF. New/updated functionality to support all of the above: - New `ShaclVueStarter`, `ShaclVueRecordsMini`, and `NodeShapeViewerMini` components - `App.vue` now dynamically loads the main entrypoint components and uses `<component :is="">` to render the component that is selected based on the `VITE_SHACLVUE_VARIANT` value - `ShaclVueStarter` uses a new `getFirstPages()` function during app startup in order to fetch the first page from the dumpthings backend for all classes. This allows getting upfront info about number of records, and some actual records, to display on the main view that shows all classes and their records. - In `ShaclVueStarter`, the `selectType` is only called once on app startup, using the `Thing` class as the selected type. - `ShaclVueStarter` has a new computed variable, `itemsByClass`, which takes all filtered records as input and groups them by class. This reactive ref is provided as the input of records to `ShaclVueRecordsMini` per class. - As with `ShaclVueRecords`, `ShaclVueRecordsMini` emits an event when the user scrolls to the bottom of the list of records. However, `ShaclVueStarter` has a new function that listens for this event and will then fetch the next page of records *for that specific class*. - `NodeShapeViewerMini` has the simplistic rendering of a record display name and menu options.

jsheunis added 1 commit

2026-05-12 22:48:44 +00:00

New approach to generating/maintaining list of records 3501e82380

The premise of this change is that the 'getInstanceItems' approach to generating the
list of displayed records (from graph data) results in bad performance. This is mainly
because 'getInstanceItems' is called many times (mainly triggered by a watcher reacting
to any/all graph changes), each time regenerating the full list of records. The proposed
new approach is based on maintaining a global list of items that is incrementally updated
when new nodes are added to or updated in the graph.

'parseTTLandDedup' is the 'ReactiveRdfDataset' class method that receives incoming TTL
strings parses it into quads, deduplicates blank nodes, and adds it to the 'N3' store.
It already provided an array with all added quads as the return value. A new utility
function, 'getUniqueRootNodes', is now used to determine the unique named nodes that are
included in the added quads. This allows saying: 'these are the records that were just
added or updated' in the graph store. 'parseTTLandDedup' now returns both the added quads
and the array of unique added/updated record IRIs, and downstream functionality using it
has been updated to access its outputs correctly. A new 'ReactiveRdfDataset' class method
'emitAddedRecords(records)' is also added, which can be called with an array of named node
(i.e. record) IRIs and which uses 'dispatchEvent' to emit the new 'recordsChanged' event,
for which listeners can be added. 'emitAddedRecords' is now used in multiple places in
order to emit records that have just been added/updated (because of this, the previous
calls to 'rdfDS.triggerReactivity' become redundant):
- In the 'useData' composable, inside 'fetchRdfData' right after 'parseTTLandDedup' returns
  the list of unique added/updated records. This happens downstream for every call to
  'fetchFromService', which is the standard way in which 'shacl-vue' gets backend data in
  response to user interaction.
- In the 'useWizard' composable: here we only call 'emitAddedRecords' if the context is not
  '_record' because it means there is no open form (which would need to process added quads
  in its own way); it means the wizard immediately adds all quads to the graph.
- In the 'FormEditor' component when the user saved the record, i.e. in the 'saveForm'
  function. Here it is called specifically when the record is a PID record (i.e. not a
  blank node).

Now, onto the algorithmic change for 'getInstanceItems', which is done in the 'useRecords'
composable. As mentioned, we now maintain a global list of items ('recordItemsAll' and
'recordItemsByClass') that is incrementally updated when new nodes are added to or updated
in the graph. This composable handles the new process as follows:
- adds an event listener to 'rdfDS' for the new 'recordsChanged' event, which calls the new
  'enqueueChanges' function
- 'enqueueChanges' adds all added/updated records to a queue, and runs the code to process
  the queue
- 'processQueue' will keep on running as long as there are records in the queue, and will
  process records in batches of 10 by calling the 'updateRecordItem' function per record
- 'updateRecordItem' runs all the code that used to be run per item by 'getInstanceItems',
  and then adds the processed record to the 'recordItemsAll' and 'recordItemsByClass'
  reactive variables
- any further computations on the global lists, such as filtering based on user-typed text,
  are done via computed refs that depend on the above reactive refs. Important new computed
  refs include 'filteredRecordItemsAll' and 'filteredRecordItemsByClass' which filters all
  records and record per class based on search text, as well as
  'filteredRecordItemsForClassWithSubclassItems' which filters records per class while also
  including all records of a class's subclasses, which is required for the existing
  'priority_classes'/'include_subclasses' config feature.

The main 'ShaclVue' component has been updated to use the adjusted 'useRecords' composable
correctly, specifically passing the new filtered computed refs to the 'ShaclVueRecords'
component, which has updated props 'filteredRecords' and 'fetchedItemCount' to handle the
new composable structure.

Also, as part of cleanup several previous computed refs were moved as functions to the
'useShapes' composable:
- 'getIdFilteredNodeShapeNames'
- 'getNoEditClassList'
- 'getFilteredNodeShapeNames'
- 'getPriorityFilteredNodeShapeNames'
- 'getOrderedNodeShapeNames'
- 'getAllClassItems'
These are now called on app startup during config processing.

Lastly, the 'useWizard' composable has been updated to include a new feature needed by the
upcoming kickstarter component. This feature allows a wizard to be shown not specifically
for a record, nor specifically for a class, but for all classes. It's goal is to allow the
configuration of very generic wizards that would be applicable for all classes. The config
spec for it's selection is as follows (with 'AddRecordWizard' being an example wizard name):

wizard_editor_selection:
  _classes:
    - AddRecordWizard

To allow distinction of the applicable class IRI when the associated wizard template is
processed, the class IRI is now made part of the wizard data by default in 'handleWizardSave'
(accessible as 'class_uri'). Also 'showWizardGroup' and 'setupWizards' functions are updated
to handle the new 'wizard_editor_selection' key.

jsheunis changed title from ~~New approach to generating/maintaining list of records and new ShaclVueStarter~~ to WIP: New approach to generating/maintaining list of records + ShaclVueStarter feature

2026-05-12 22:50:53 +00:00

jsheunis added 1 commit

2026-05-13 09:55:11 +00:00

Introduce new 'ShaclVueStarter' app variant 7cdde78878

This work is in response to the identified need for kick-starting the user-based entry of domain-specific
knowledge into a 'shacl-vue' metadata system, so that a critical mass for continued use of the system is
available soon after deploying it for the first time. The new 'ShaclVueStarter' component is designed to
encourage users to enter minimal records in a quick, intuitive, and interactive way. It has the following
features:

1. It serves as a new variant of the main entrypoint of a 'shacl-vue' deployment. This requires the
   specification of a Vite-recognizable environment variable, 'VITE_SHACLVUE_VARIANT' that influences
   whether the application is built with the (default) 'ShaclVue' component or the new 'ShaclVueStarter'
   component. An example command for building the starter app:

   VITE_SHACLVUE_VARIANT=starter npm run build:app

   If the environment variable is not specified, the default application will be built.

2. Instead of making only a single class selectable at a time, 'ShaclVueStarter' shows all record types
   (aka classes), and many records per class. The user can scroll down the main view through all classes,
   and relevant records can be scrolled within each class 'card'. The left-hand-side pane now shows the
   list of classes with added checkboxes, which allows including/exluding classes from being displayed in
   the main view.

3. A 'ShaclVueStarter' deployment uses the new '_classes' option of the 'wizard_selection' configuration
   to specify a wizard that will be shown for all classes. This allows the use of a generic 'add new record'
   wizard that only asks for a display name and description, with example config:

  AddRecordWizard:
    name: Add New Record Wizard
    tooltip: Add a new record
    icon: mdi-plus-circle-outline
    description: Add the display name and description and then hit *Save*
    inputs:
      - prop: name
        name: Display name
        description: The display name of the new record
        type: text
        placeholder: 'My record title'
        required: true
      - prop: description
        name: Description
        description: The description of the new record
        type: text-paragraph
        placeholder: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod...'
    template: content:add-record

4. Text search functionality provides the same UX as before, however the search now happens across all
   records, i.e. not only on a single class-level.

5. Unnecessary detail and functionality from the default 'ShaclVue' application are not available in the
   starter application. This mainly means the record viewer is minimalistic, containing just the display
   name and a mini menu button with options to edit the record (which will open the established form) and
   view the record RDF.

New/updated functionality to support all of the above:

- New 'ShaclVueStarter', 'ShaclVueRecordsMini', and 'NodeShapeViewerMini' components
- 'App.vue' now dynamically loads the main entrypoint components and uses 'component :is=' to render the
  component that is selected based on the 'VITE_SHACLVUE_VARIANT' value
- 'ShaclVueStarter' uses a new 'getFirstPages()' function during app startup in order to fetch the first
  page from the dumpthings backend for all classes. This allows getting upfront info about number of
  records, and some actual records, to display on the main view that shows all classes and their records.
- In 'ShaclVueStarter', the 'selectType' is only called once on app startup, using the 'Thing' class as
  the selected type.
- 'ShaclVueStarter' has a new computed variable, 'itemsByClass', which takes all filtered records as input
  and groups them by class. This reactive ref is provided as the input of records to 'ShaclVueRecordsMini'
  per class.
- As with 'ShaclVueRecords', 'ShaclVueRecordsMini' emits an event when the user scrolls to the bottom of
  the list of records. However, 'ShaclVueStarter' has a new function that listens for this event and will
  then fetch the next page of records for that specific class.
- 'NodeShapeViewerMini' has the simplistic rendering of a record display name and menu options.

jsheunis changed title from ~~WIP: New approach to generating/maintaining list of records + ShaclVueStarter feature~~ to New approach to generating/maintaining list of records + ShaclVueStarter feature

2026-05-13 09:55:48 +00:00

jsheunis merged commit abcc71a7d7 into main

2026-05-13 21:54:18 +00:00

jsheunis referenced this pull request from a commit

2026-05-13 21:54:19 +00:00

Merge pull request 'New approach to generating/maintaining list of records + `ShaclVueStarter` feature' (#349) from shaclvue-kickstarter into main