- When Spreadsheets Slow Down, Explore Smarter Ways to Move Forward
Hello all,
I have been venturing on quite the Excel journey the past year or so. I made a corporate spreadsheet that is approaching 500k formulas and that is starting to get serious speed issues at this point. It is 2026, so I conversed with ChatGPT several times regarding the speed issue, but realized I am way better off asking the experts here anyways.
What is the problem
So, my spreadsheet imports flat databases with specific information regarding objects that need further analysing. The imported flat databases run from say A tot CC or something, from which I probably draw about 12-15 datafields that are used for further analysis. It 'may' be more in the future. Afterwards, said data gets 'enriched' (manually) by things that aren't in the database, also because said data needs a human eye that cannot be automated. So far, so good.
Right now, each object gets analysed from several different angles. As it stands, my spreadsheet runs from A until NA or something on the Formula Page. Many columns receive data from preceding columns, that are in the turn the result of many (slightly complex) logical IF or IFS tests, many of which are nested 3 or 4 deep. Often, they work in conjunction with X.LOOKUP to retrieve values, as the columns on the formula page are not equal.
For example: A until BC on the Formula Page may analyze 150 objects, BD until DD may analyse 100 objects (from the same dataset, so narrower), and so forths. Thus a lot of X.LOOKUP is required, also because the first 'block' comes up with values that need to be found with X.LOOKUP. Also, values need to be retrieved from the flat database 'import' page with X.LOOKUP. Finally, X.LOOKUP is an insurance compared to FILTER, as I am not fully convinced that empty values in the flat database always contain a space (" ").
To get to the point
I use many IF, IFS, AND, and if need be, OR, formulas. Thinks: tens of thousands, probably in excess of 100k.
These are compounded with X.LOOKUP, or X.LOOKUP gets used copiously without those. Here too, think tens of thousands.
These formulas are - as much as possible - in array format, even though I find it controversial to do that as I consider how it can create a chain of updates throughout the spreadsheet.
'Dependencies' is the name of the game, with one object receiving many possible alterations / adjustments due to manual input data, for which the spreadsheet needs to provide.
Right now, when I update a value, it may take up to 4 seconds to update the spreadsheet, which is already beyond the annoyance point for me. This leads me to these (hopefully) simple questions:
Is it smart to use array formulas, knowing that each thing I change should only impact that one object line (for example, row 488) and none other? It is important to mention that object 1 does not influence object 488, or any other. Any manual data field only effects the object in the row it is in. In my mind, array formulas do not make sense in that regard, as it can result in a cascade of updates, but apparantly array formulas are 'way more efficient'.
Is use of a VBA library the way to go to reduce lag and create more of an instant spreadsheet again? I am not able to code in VBA yet, but I am in the slow process of learning it regardless.
Alternatively: should I use LET whenever a repeated lookup is needed in the same formula?
Really looking for to your answers!
submitted by /u/EvolvedRevolution
[link] [comments]
- Your spreadsheets are stuck in the 1980s. Let's explore what comes next.
Hi everyone,
I’m a CSIE student in Taiwan, and I’ve spent the last semester obsessing over why "data organization" still feels like manual labor. We have incredible processing power, yet most of us are still stuck in the "Shovel Era", manually digging through rows, fixing broken VLOOKUPs, and praying our CSV imports don't break.
I wanted to share three specific "Excel Pains" I’ve been researching while building my own organizer, and I’d love to hear if you’ve found better ways to handle them:
1. The "Syntax Trap" vs. Human Intent
Most people spend 80% of their time worrying about where the comma goes in a nested IF statement and only 20% on what the data actually means. I believe we are moving toward a "Semantic Era" where the computer should understand that "March 26" and "03/26/26" are the same thing without us writing a regex script.
2. The "Final_v2_FINAL_ActuallyFinal.xlsx" Nightmare
File organization usually falls apart because our tools don't track the lineage of data. When we move from a messy raw file to a "clean" one, we lose the context of the original. I've been experimenting with building a "Tractor" for this—a system where the AI maintains a "Kanban" of data states so you can see the evolution of your project visually.
3. The 2FA/Security Gap in Spreadsheets
We put our lives into Excel files, but standard spreadsheets are notoriously easy to leak or lose. I’ve been implementing 2FA data protection into my workflow because "Data Organization" shouldn't just be about sorting; it should be about stewardship.
The Project: Dxtreame Organizer To solve these, I’ve been building Dxtreame Organizer. It’s an AI-driven tool meant to bridge that gap between messy raw data and structured, formula-ready Excel sheets.
Current Progress: I've got the AI sorting engine running, 2FA protection live, and I'm currently designing a graph-view to replace the "wall of numbers" we usually stare at.
The Goal: I’m currently fundraising as an international student to scale the infrastructure. My vision is to get rid of the "reason to learn syntax" entirely, so we can focus on the Vision instead of the Code.
I’m looking for brutally honest feedback:
What is the one thing in Excel that makes you want to throw your laptop out a window?
If an AI could "auto-clean" your files, what is the one thing you would NEVER trust it to do alone?
Thanks for reading, I'm looking forward to the "logic vs. automation" debate in the comments!
submitted by /u/Dxxx101
[link] [comments]
- Discover a simpler path through complex data tasks with AI-driven clarity.
I'm going to describe a situation as specifically as I can. I am curious what people would do in this situation, I worry that I complicate things for myself. I'm describing the whole task as it was described to me and then as I discovered it.
Ultimately, I'm here to ask you, what do you do, and how long does it take you to do it?
I started a new role this month, I am new to advertising modeling methods like mmm, so I am reading a lot about how to apply the methods specific to mmm in R and python, I use VScode, I don't have a github copilot license, I get to use copilot through windows office license. Although this task did not involve modeling, I do want to ask about that kind of task another day if this goes over well.
The task
5, excel sheets are to be provided. You are told that this is a clients data that was given to another party for some other analysis and augmentation. This is a quality assurance task. The previous process was as follows;
the data
the data structure: 1 workbook per industry for 5 industries
4 workbooks had 1 tab, 1 workbook had 3 tabs
each tab had a table that had a date column in days, 2 categorical columns advertising_partner, line_of_business and at least 2 numeric columns per work book.
some times data is updated from our side and the partner has to redownload the data and reprocess and share again
the process
this is done once per client, per quarter (but it's just this client for now)
open each workbook
navigate to each tab
the data is in a "controllable" table
bing bing
home home
impressions spend partner dropdown line of business dropdown
where bing and home are controlled with drop down toggles, with a combination of 3-4 categories each.
compare with data that is to be downloaded from a tableau dashboard
end state: the comparison of the metrics in tableau to the excel tables to ensure that "the numbers are the same"
the categories presented map 1 to 1 with the data you have downloaded from tableau
aggregate the data in a pivot table, select the matching categories, make sure the values match
additional info about the file
the summary table is a complicated sumproduct look up table against an extremely wide table hidden to the left. the summary table can start as early as AK and as late as FE.
there are 2 broadly different formats of underlying data in the 5 notebooks, with small structure differences between the group of 3.
in the group of 3
the structure of this wide table is similar to the summary table with categories in the column headers describing the metric below it. but with additional categories like region, which is the same value for every column header. 1 of these tables has 1 more header category than the other 2
the left most columns have 1 category each, there are 3 date columns for day, quarter.
REGION USA USA USA
PARTNER bing bing google
LOB home home auto
impressions spend ...etc
date quarter impressions spend ...etc
2023-01-01 q1 1 2 ...etc
2023-01-02 q1 3 4 ...etc
in the group of 2
the left most categories are actually the categorical headers in the group of 3, and the metrics, the values in each category mach
the dates are now the headers of this very wide table
the header labels are separated from the start of the values by 1 column
there is an empty row immediately below the final row for column headers.
date Label 2023-01-01 2023-01-02
year 2023 2023
quarter q1 q1
blank row
REGION PARTNER LOB measure
blank row
US bing home impressions 1 3
US bing home spend 2 4
US google auto ...etc ...etc ... etc
The question is, what do you do, and how long does it take you to do it?
I am being honest here, I wrote out this explaination basically in the order in which I was introduced to the information and how I discovered it. (Oh it's easy if it's all the same format even if it's weird, oh there are 2-ish different formatted files)
the meeting of this task ended at 11:00AM. I saw this copy paste manual etl project and I simply didn't want to do it. So I outlined my task by identifying the elements of the table, column name ranges, value ranges, stacked / pivoted column ranges, etc... for an R script to extract that data. by passing the ranges of that content to an argument make_clean_table(left_columns="B4:E4", header_dims=c(..etc)) and functions that extract that convert that excel range into the correct position in the table to extract that element. Then the data was transformed to create a tidy long table.
the function gets passed once per notebook extracting the data from each worksheet, building a single table with the columns for the workbook industry, the category in the tab, partner, line of business, spend, impressions, etc...
IMO; ideally (if I have to check their data in excel that is), I'd like the partner to redo their report so that I received a workbook with the underlying data in a traditionally tabular form and their reporting page to use power query and table references and not cell ranges and formula.
submitted by /u/TheTresStateArea
[link] [comments]