vikunja
/
vikunja
6
43
Fork 71
Code Issues 150 Pull Requests 15 Packages Releases 32 Activity

addListDetails was not optimized to build unique list of list owners #1189

Closed
k2s wants to merge 2 commits from k2s/api:k2s-addListDetails-too-many into main
pull from: k2s/api:k2s-addListDetails-too-many
merge into: vikunja:main
Conversation 2 Commits 2 Files Changed 1 +5 -1
k2s commented 2022-06-19 19:02:17 +00:00
Contributor
Copy Link

solves one of multiple problems related to #1188

solves one of multiple problems related to #1188
k2s added 1 commit 2022-06-19 19:02:19 +00:00
continuous-integration/drone/pr Build is failing Details
d8b23ceb9f addListDetails was not optimized to build unique list of list owners 
solves one of multiple problems related to #1188
k2s added 1 commit 2022-06-19 19:04:23 +00:00
continuous-integration/drone/pr Build is passing Details
d7827d8c49 Merge branch 'main' into k2s-addListDetails-too-many
konrad commented 2022-07-02 12:18:17 +00:00
Owner
Copy Link

This is a bit more complicated.

MySQL and Microsoft Sql server (I think?) will optimize huge where in() queries with duplicate values. Postgres or SQLite won't do that and SQLite has an upper limit of how much values it can handle (as you probably already discovered).

I've asked about this on Stackoverflow: https://stackoverflow.com/q/62392660/10924593
For postgres it seems like it can handle these huge lists of values without provlems so it won't bother to optimize them.

Now because there's a lot of cases where we're using these kinds of queries this should ideally go into the orm. In fact, I started a PR some time ago do o just that: https://gitea.com/xorm/builder/pulls/76
I think we should continue that one instead of manually adding the cleanup everywhere in Vikunja. Maybe it'd worth it to modify that PR so that it would only optimize for SQLite since this only seems to be a problem with SQLite and not other DMBS.

This is a bit more complicated. MySQL and Microsoft Sql server (I think?) will optimize huge `where in()` queries with duplicate values. Postgres or SQLite won't do that and SQLite has an upper limit of how much values it can handle (as you probably already discovered). I've asked about this on Stackoverflow: https://stackoverflow.com/q/62392660/10924593 For postgres it seems like it can handle these huge lists of values without provlems so it won't bother to optimize them. Now because there's a lot of cases where we're using these kinds of queries this should ideally go into the orm. In fact, I started a PR some time ago do o just that: https://gitea.com/xorm/builder/pulls/76 I think we should continue that one instead of manually adding the cleanup everywhere in Vikunja. Maybe it'd worth it to modify that PR so that it would only optimize for SQLite since this only seems to be a problem with SQLite and not other DMBS.
konrad commented 2023-10-11 08:55:47 +00:00
Owner
Copy Link

The PR in xorm was merged, I think we can close this now.

The PR in xorm was merged, I think we can close this now.
konrad closed this pull request 2023-10-11 08:55:52 +00:00
All checks were successful
continuous-integration/drone/pr Build is passing
Required
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
dependencies

   
duplicate

This issue or pull request already exists

  
help wanted

Need some help

  
invalid

Something is wrong

  
kind/bug

Something is not working

  
kind/feature

New feature

  
needs reproduction

   
question

More information is needed

  
security

   
wontfix

This won't be fixed

No Label
dependencies
duplicate
help wanted
invalid
kind/bug
kind/feature
needs reproduction
question
security
wontfix
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: vikunja/vikunja#1189
No description provided.
Delete Branch "k2s/api:k2s-addListDetails-too-many"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?