- Slow load times slow your workflow, not your expertise
Good evening,
I work in a food production plant in Shipping and Receiving.
We have had Microsoft Forms for entering in daily cases produced, cases shipped, and a separate form for doing time studies on trucks that come in, how long to load or unload said truck, and when they leave.
I have had a manual workbook to fill in all of this data basically again (this information gets entered into these daily reports we fill out in our Microsoft forms) but to organize it into an easy daily report to give us truck In to Out averages, loading time averages, cases produced vs what was scheduled to produce, etc.. A big issue I have had with this manual data entry workbook, which are done month by month, is the amount of formulas which I have in it..(multiplying cases by item number to give us weight and how many skids, calculating our scheduled amount to produce against what's actually produced, giving percentages, many conditional formatted cells to easily show if we are in the green or red, etc.)
Now my boss has always wanted a workbook to do what my manual workbook does but to grab the data from the Excel workbook that these Microsoft forms load the data into. The problem before was we had two separate Microsoft forms for daily cases produced/shipped and the one for our time studies.
But I went ahead and made one form which would do both.
I was able to copy over many sheets and formulas from my manual workbook into the Excel spreadsheet that loads in the data from this Microsoft Form. My boss really wants it to work indefinitely..
The problem I am encountering which I was afraid of, is the amount of formulas in this one workbook is way too much for a computer to handle. Changing 1 thing results in it needing to calculate a thread for like 20-30 minutes (like with the manual excel spreadsheet, the manual processor has been set to 1).
Am I just going about this all wrong? Is there a better way to grab the data from this form that isn't going to overload a computer? Do I make separate workbooks pulling from this form's Excel workbook and just keep the daily report with the initial Microsoft Form workbook (but then would those workbooks update automatically as well?)
I imagine there is a way to achieve what my boss is wanting, but my experience with Excel is only so advanced. I'm aware there are other programs or other tools of excel, and that is why I came onto this subreddit for advice. Please help me 🙇🏻♂️
submitted by /u/maverickrose
[link] [comments]
- Discover a simpler path through complex data tasks with AI-driven clarity.
I'm going to describe a situation as specifically as I can. I am curious what people would do in this situation, I worry that I complicate things for myself. I'm describing the whole task as it was described to me and then as I discovered it.
Ultimately, I'm here to ask you, what do you do, and how long does it take you to do it?
I started a new role this month, I am new to advertising modeling methods like mmm, so I am reading a lot about how to apply the methods specific to mmm in R and python, I use VScode, I don't have a github copilot license, I get to use copilot through windows office license. Although this task did not involve modeling, I do want to ask about that kind of task another day if this goes over well.
The task
5, excel sheets are to be provided. You are told that this is a clients data that was given to another party for some other analysis and augmentation. This is a quality assurance task. The previous process was as follows;
the data
the data structure: 1 workbook per industry for 5 industries
4 workbooks had 1 tab, 1 workbook had 3 tabs
each tab had a table that had a date column in days, 2 categorical columns advertising_partner, line_of_business and at least 2 numeric columns per work book.
some times data is updated from our side and the partner has to redownload the data and reprocess and share again
the process
this is done once per client, per quarter (but it's just this client for now)
open each workbook
navigate to each tab
the data is in a "controllable" table
bing bing
home home
impressions spend partner dropdown line of business dropdown
where bing and home are controlled with drop down toggles, with a combination of 3-4 categories each.
compare with data that is to be downloaded from a tableau dashboard
end state: the comparison of the metrics in tableau to the excel tables to ensure that "the numbers are the same"
the categories presented map 1 to 1 with the data you have downloaded from tableau
aggregate the data in a pivot table, select the matching categories, make sure the values match
additional info about the file
the summary table is a complicated sumproduct look up table against an extremely wide table hidden to the left. the summary table can start as early as AK and as late as FE.
there are 2 broadly different formats of underlying data in the 5 notebooks, with small structure differences between the group of 3.
in the group of 3
the structure of this wide table is similar to the summary table with categories in the column headers describing the metric below it. but with additional categories like region, which is the same value for every column header. 1 of these tables has 1 more header category than the other 2
the left most columns have 1 category each, there are 3 date columns for day, quarter.
REGION USA USA USA
PARTNER bing bing google
LOB home home auto
impressions spend ...etc
date quarter impressions spend ...etc
2023-01-01 q1 1 2 ...etc
2023-01-02 q1 3 4 ...etc
in the group of 2
the left most categories are actually the categorical headers in the group of 3, and the metrics, the values in each category mach
the dates are now the headers of this very wide table
the header labels are separated from the start of the values by 1 column
there is an empty row immediately below the final row for column headers.
date Label 2023-01-01 2023-01-02
year 2023 2023
quarter q1 q1
blank row
REGION PARTNER LOB measure
blank row
US bing home impressions 1 3
US bing home spend 2 4
US google auto ...etc ...etc ... etc
The question is, what do you do, and how long does it take you to do it?
I am being honest here, I wrote out this explaination basically in the order in which I was introduced to the information and how I discovered it. (Oh it's easy if it's all the same format even if it's weird, oh there are 2-ish different formatted files)
the meeting of this task ended at 11:00AM. I saw this copy paste manual etl project and I simply didn't want to do it. So I outlined my task by identifying the elements of the table, column name ranges, value ranges, stacked / pivoted column ranges, etc... for an R script to extract that data. by passing the ranges of that content to an argument make_clean_table(left_columns="B4:E4", header_dims=c(..etc)) and functions that extract that convert that excel range into the correct position in the table to extract that element. Then the data was transformed to create a tidy long table.
the function gets passed once per notebook extracting the data from each worksheet, building a single table with the columns for the workbook industry, the category in the tab, partner, line of business, spend, impressions, etc...
IMO; ideally (if I have to check their data in excel that is), I'd like the partner to redo their report so that I received a workbook with the underlying data in a traditionally tabular form and their reporting page to use power query and table references and not cell ranges and formula.
submitted by /u/TheTresStateArea
[link] [comments]
- When Spreadsheets Slow Down, Explore Smarter Ways to Move Forward
Hello all,
I have been venturing on quite the Excel journey the past year or so. I made a corporate spreadsheet that is approaching 500k formulas and that is starting to get serious speed issues at this point. It is 2026, so I conversed with ChatGPT several times regarding the speed issue, but realized I am way better off asking the experts here anyways.
What is the problem
So, my spreadsheet imports flat databases with specific information regarding objects that need further analysing. The imported flat databases run from say A tot CC or something, from which I probably draw about 12-15 datafields that are used for further analysis. It 'may' be more in the future. Afterwards, said data gets 'enriched' (manually) by things that aren't in the database, also because said data needs a human eye that cannot be automated. So far, so good.
Right now, each object gets analysed from several different angles. As it stands, my spreadsheet runs from A until NA or something on the Formula Page. Many columns receive data from preceding columns, that are in the turn the result of many (slightly complex) logical IF or IFS tests, many of which are nested 3 or 4 deep. Often, they work in conjunction with X.LOOKUP to retrieve values, as the columns on the formula page are not equal.
For example: A until BC on the Formula Page may analyze 150 objects, BD until DD may analyse 100 objects (from the same dataset, so narrower), and so forths. Thus a lot of X.LOOKUP is required, also because the first 'block' comes up with values that need to be found with X.LOOKUP. Also, values need to be retrieved from the flat database 'import' page with X.LOOKUP. Finally, X.LOOKUP is an insurance compared to FILTER, as I am not fully convinced that empty values in the flat database always contain a space (" ").
To get to the point
I use many IF, IFS, AND, and if need be, OR, formulas. Thinks: tens of thousands, probably in excess of 100k.
These are compounded with X.LOOKUP, or X.LOOKUP gets used copiously without those. Here too, think tens of thousands.
These formulas are - as much as possible - in array format, even though I find it controversial to do that as I consider how it can create a chain of updates throughout the spreadsheet.
'Dependencies' is the name of the game, with one object receiving many possible alterations / adjustments due to manual input data, for which the spreadsheet needs to provide.
Right now, when I update a value, it may take up to 4 seconds to update the spreadsheet, which is already beyond the annoyance point for me. This leads me to these (hopefully) simple questions:
Is it smart to use array formulas, knowing that each thing I change should only impact that one object line (for example, row 488) and none other? It is important to mention that object 1 does not influence object 488, or any other. Any manual data field only effects the object in the row it is in. In my mind, array formulas do not make sense in that regard, as it can result in a cascade of updates, but apparantly array formulas are 'way more efficient'.
Is use of a VBA library the way to go to reduce lag and create more of an instant spreadsheet again? I am not able to code in VBA yet, but I am in the slow process of learning it regardless.
Alternatively: should I use LET whenever a repeated lookup is needed in the same formula?
Really looking for to your answers!
submitted by /u/EvolvedRevolution
[link] [comments]