155 lines
19 KiB
Plaintext
155 lines
19 KiB
Plaintext
|
+
|
|||
|
|
Î×i]; ã óX € R t ^ RIHt ^ RIt^ RIt^ RIHt ^ RIHt ^ RI t
|
|||
|
|
^ RI H t H
|
|||
|
|
t
|
|||
|
|
HtHtHtHtHt ]P& ! ]4 t]P, ! R]RR7 R R
|
|||
|
|
ltR R ltR
|
|||
|
|
R ltR R ltR R lt ]3R R llt ]3R R llt ]3R R llt ]]]3R R llt R R lt!R R lt"R# ) uØ
|
|||
|
|
data_loader.py — Load and normalise the three SHEQ data sources.
|
|||
|
|
|
|||
|
|
Each loader returns a pandas DataFrame with normalised column names
|
|||
|
|
(defined in config.py) so that downstream analysis code is insulated
|
|||
|
|
from changes to the source file schema.
|
|||
|
|
|
|||
|
|
Public API
|
|||
|
|
----------
|
|||
|
|
load_events(filepath) -> pd.DataFrame
|
|||
|
|
load_safety_energy(filepath) -> pd.DataFrame
|
|||
|
|
load_llc_data(filepath) -> pd.DataFrame
|
|||
|
|
load_all(events_path, se_path, llc_path) -> dict[str, pd.DataFrame]
|
|||
|
|
)Ú annotationsN)ÚPath)ÚOptional)ÚEVENTS_COL_MAPÚ
|
|||
|
|
SE_COL_MAPÚ LLC_COL_MAPÚMODULE_TYPE_LABELSÚ EVENTS_FILEÚSAFETY_ENERGY_FILEÚLLC_FILEÚignoreÚopenpyxl)ÚcategoryÚmodulec ó( € V ^8„ d QhRRRRRRRR/# ) é Údfú pd.DataFrameÚ
|
|||
|
|
candidatesz list[str]ÚkeyÚstrÚreturnz
|
|||
|
|
Optional[str]© )Úformats "Údata_loader.pyÚ __annotate__r - s( € ÷ ñ �\ð ¨yð ¸sð À}ñ ó c ól € V F pW0P 9 g K Vu # \ P RW!4 R# )z=Return the first candidate column that exists in df, or None.z%Column key '%s' not found (tried: %s)N)ÚcolumnsÚlogÚdebug)r r r Úcs &&& r Ú _resolve_colr" - s. € ã
|
|||
|
|
ˆØ —
|
|||
|
|
‘
|
|||
|
|
Ž?ØŠHñ ô ‡I�IÐ5°sÔGÙ r c ó € V ^8„ d QhRRRR/# ©r Úseriesú pd.Seriesr r )r s "r r r 6 s € ÷ "ñ "˜ð " yñ "r c óò € \ P P P V 4 ' d6 V P P
|
|||
|
|
e V P P
|
|||
|
|
R4 # T # R pV P V4 # )zÔ
|
|||
|
|
Parse a date series that may contain:
|
|||
|
|
- ISO strings "2024-01-15"
|
|||
|
|
- Long-form strings "Monday, 15 January 2024"
|
|||
|
|
- Excel datetime objects
|
|||
|
|
Returns a tz-naive datetime64 series; unparseable values become NaT.
|
|||
|
|
Nc óÄ € \ P ! V 4 ' d \ P # \ V 4 P 4 pR V9 d\ \ VP
|
|||
|
|
R 4 ^ , P
|
|||
|
|
4 4 ^8X d( VP
|
|||
|
|
R ^4 ^, P 4 p \ P ! VRR7 # \ d \ P u # i ; i)Ú,T)Údayfirst) ÚpdÚisnaÚNaTr ÚstripÚlenÚsplitÚ to_datetimeÚ Exception)ÚvalÚss & r Ú
|
|||
|
|
_parse_oneÚ _parse_dates.<locals>._parse_oneA s› € Ü
|
|||
|
|
�7Š7�3�<Š<Ü—6‘6ˆMÜ �‹H�N‰NÓ ˆà �!Œ8œ˜AŸG™G C›L¨�O× 1Ñ 1Ó 3Ó4¸Ô9Ø—‘˜˜Q“ Õ"×(Ñ(Ó*ˆAð Ü—>’> !¨dÔ3Ð 3øÜô Ü—6‘6ŠMð ús Â)C ÃC à C )r+ ÚapiÚtypesÚis_datetime64_any_dtypeÚdtÚtzÚ tz_localizeÚmap)r% r5 s & r Ú _parse_datesr> 6 sZ € ô
|
|||
|
|
‡v�v‡|�|×+Ñ+¨F×3Ò3Ø.4¯i©i¯l©lÒ.Fˆv�y‰y×$Ñ$ TÓ*ÐRÈFÐRò
|
|||
|
|
ð �:‰:�jÓ !Ð!r c ó$ € V ^8„ d QhRRRRRR/# )r r r Úcol_mapzdict[str, list[str]]r r )r s "r r r P s" € ÷ ñ ˆ|ð Ð&:ð ¸|ñ r c óÐ € V P 4 pVP 4 FA w r4\ WV4 pVe W0P 9 d
|
|||
|
|
W, W#& K1 Vf K7 W, W#& KC V# )a:
|
|||
|
|
Build a new DataFrame with normalised column names.
|
|||
|
|
|
|||
|
|
For each key in col_map, find the first matching source column and
|
|||
|
|
rename it. Columns not mentioned in col_map are dropped. The
|
|||
|
|
original source columns are preserved under their original names as
|
|||
|
|
well, allowing callers to access additional fields if needed.
|
|||
|
|
)ÚcopyÚitemsr" r )r r@ ÚresultÚ norm_namer Úsrcs && r Ú_remaprG P s\ € ð �W‰W‹Y€FØ!(§ ¡ ¦ Ñ ˆ ܘ2¨9Ó5ˆØ Š?˜y·
|
|||
|
|
±
|
|||
|
|
Ô :Ø "¥ˆFÓ Ø
|
|||
|
|
Œ_Ø "¥ˆFÓ ñ "1ð €Mr c ó € V ^8„ d QhRRRR/# )r r% r& r Úfloatr )r s "r r r d s € ÷ ñ �yð Uñ r c ó> € V P 4 P 4 # )u/ Return fraction of null / empty values (0–1).)r, Úmean)r% s &r Ú
|
|||
|
|
_null_raterL d s € à �;‰;‹=× Ñ Ó Ð r c ó$ € V ^8„ d QhRRRRRR/# )r r r Úlabelr r Údictr )r s "r r r i s! € ÷ ñ � ð cð ¨dñ r c óª € RVR\ V 4 R\ V P 4 R\ V P R\ P
|
|||
|
|
! RR7 4 4 /# )z1Return a simple quality profile dict for logging.ÚsourceÚrowsÚcolsÚ
|
|||
|
|
date_nullsÚdateÚobject©Údtype)r/ r rL Úgetr+ ÚSeries)r rN s &&r Ú_profiler[ i sG € ð �%Ø”�B“Ø”�B—J‘J“Ø”j §¡¨´· ² ÀÔ0IÓ!JÓKð ð r c ó € V ^8„ d QhRRRR/# ©r Úfilepathr r r r )r s "r r r w s € ÷ 2ñ 2˜#ð 2° ñ 2r c ó € \ V 4 pVP 4 ' g \ RV 24 h\ P RV 4 \
|
|||
|
|
P ! V 4 p\ P ! R.VP O5! \ V\ 4 p\ VR, 4 VR& \ V4 pVP R.R7 P 4 p\ V4 V8 d' \ P RV\ V4 ,
|
|||
|
|
4 VR, P P VR& VR, P P" VR& VR, P P% R 4 VR
|
|||
|
|
& VR, P P' 4 VR & R F“ pWSP( 9 g K W5, P+ \, 4 P, P/ 4 W5& W5, P1 R
|
|||
|
|
\
|
|||
|
|
P2 R\
|
|||
|
|
P2 R\
|
|||
|
|
P2 /4 W5& K• \5 VR4 p\ P RVR, R V9 d5 \7 VR , P 4 P9 4 4 4 V# R4 V# )a‘
|
|||
|
|
Load Events.xlsx and return a normalised DataFrame.
|
|||
|
|
|
|||
|
|
Normalised columns (see EVENTS_COL_MAP):
|
|||
|
|
date, event_type, consequence, status, business_unit, project,
|
|||
|
|
location, crp, root_cause_cat, root_cause_sub, injury_class,
|
|||
|
|
body_part, brief_desc, event_desc, days_to_enter, event_lag,
|
|||
|
|
report_lag, investigation_done, hipo, critical_event
|
|||
|
|
|
|||
|
|
Also adds:
|
|||
|
|
year, month, year_month (Period[M])
|
|||
|
|
zEvents file not found: zLoading Events from %sõ Raw shape: %s rows × %s colsrU ©Úsubsetú# Dropped %d rows with missing dateÚyearÚmonthÚMÚ
|
|||
|
|
year_monthÚdowÚ
|
|||
|
|
business_unitÚnanÚNoneÚ ÚEventsz Loaded %d events | BUs: %srR Ú?)Ú
|
|||
|
|
event_typeÚ consequenceri ÚprojectÚroot_cause_catÚ injury_class) r ÚexistsÚFileNotFoundErrorr Úinfor+ Ú
|
|||
|
|
read_excelÚshaperG r r> r/ ÚdropnarB Úwarningr: rd re Ú to_periodÚday_namer Úastyper r. ÚreplaceÚNAr[ ÚlistÚunique)r^ ÚpathÚrawr Ún_beforeÚcolÚprofiles & r Ú load_eventsr‡ w s÷ € ô �‹>€DØ �;‰;�=Š=Ü Ð"9¸(¸Ð DÓEÐEä‡H�HÐ
|
|||
|
|
% xÔ0Ü
|
|||
|
|
�-Š-˜Ó
|
|||
|
|
!€C܇H‚HÐ
|
|||
|
|
.Ð;°·±Ô;ä �”^Ó $€Bô ˜b �jÓ)€B€v�Jô �2‹w€HØ �‰˜6˜(ˆÓ #× (Ñ (Ó *€BÜ
|
|||
|
|
ˆ2ƒw�ÔÜ � ‰ Ð9¸8ÄcÈ"ÃgÕ;MÔNð ˜&•z—}‘}×)Ñ)€B€v�Jؘ&•z—}‘}×*Ñ*€B€w�Kؘ&•z—}‘}×.Ñ.¨sÓ3€B€|Ñؘ&•z—}‘}×-Ñ-Ó/€B€u�Ió2ˆà —*‘*Ö Ø•g—n‘n¤SÓ)×-Ñ-×3Ñ3Ó5ˆB‰GØ•g—o‘o u¬b¯e©e°V¼R¿U¹UÀBÌÏÉÐ&NÓOˆB‹Gñ 2ô �r˜8Ó$€G܇H�HÐ
|
|||
|
|
+Ø
|
|||
|
|
�V�_Ø<KÈrÔ<QŒT�"�_Õ%×,Ñ,Ó.×5Ñ5Ó7Ó
|
|||
|
|
8ô\ð €Ið X[ô\ð €Ir c ó € V ^8„ d QhRRRR/# r] r )r s "r r r ° s € ÷ =ñ = ð =¸lñ =r c óÊ € \ V 4 pVP 4 ' g \ RV 24 h\ P RV 4 \
|
|||
|
|
P ! V 4 p\ P ! R.VP O5! \ V\ 4 p\ VR, 4 VR& \ V4 pVP R.R7 P 4 p\ V4 V8 d' \ P RV\ V4 ,
|
|||
|
|
4 VR, P \ 4 P# VP% R\
|
|||
|
|
P& ! RR 7 4 4 VR
|
|||
|
|
& VR, P( P* VR & VR, P( P, VR & VR, P( P/ R
|
|||
|
|
4 VR& R F“ pWSP0 9 g K W5, P3 \4 4 P4 P7 4 W5& W5, P9 R\
|
|||
|
|
P: R\
|
|||
|
|
P: R\
|
|||
|
|
P: /4 W5& K• R F5 pWSP0 9 g K \
|
|||
|
|
P< ! W5, RR7 W5& K7 \ P R\ V4 R
|
|||
|
|
V9 d, VR
|
|||
|
|
, P? 4 PA 4 4 V# R4 V# )u
|
|||
|
|
Load Safety_Energy.xlsx and return a normalised DataFrame.
|
|||
|
|
|
|||
|
|
Safety Energy is the combined analytical domain covering all leading
|
|||
|
|
activity types: LLC (Leader Learning Conversations), CCC (Critical
|
|||
|
|
Control Checks), and OCC (Operational Control Checks).
|
|||
|
|
|
|||
|
|
Normalised columns (see SE_COL_MAP):
|
|||
|
|
date, module_name, module_prefix, module_type, activity_type
|
|||
|
|
(short label: LLC/CCC/OCC), leader, business_unit, project,
|
|||
|
|
location, at_risk_aspects, total_questions, actions, atl_actions,
|
|||
|
|
at_risk_crp, llc_topic, at_risk_obs, positive_obs, participants
|
|||
|
|
|
|||
|
|
Also adds:
|
|||
|
|
year, month, year_month (Period[M])
|
|||
|
|
activity_type — shortened label from MODULE_TYPE_LABELS
|
|||
|
|
z Safety Energy file not found: z Loading Safety Energy from %sr` rU ra rc Ú module_typer rW Ú
|
|||
|
|
activity_typerd re rf rg rj rk rl Úcoerce©Úerrorsz" Loaded %d activities | types: %srn )ri rq Úleaderr‹ )Úat_risk_aspectsÚtotal_questionsÚactionsÚ atl_actions)!r rt ru r rv r+ rw rx rG r r> r/ ry rB rz r= r ÚfillnarY rZ r: rd re r{ r r} r r. r~ r Ú
|
|||
|
|
to_numericÚ value_countsÚto_dict©r^ r‚ rƒ r r„ r… s & r Úload_safety_energyr™ ° s4 € ô$ �‹>€DØ �;‰;�=Š=Ü Ð"@ÀÀ
|
|||
|
|
Ð KÓLÐLä‡H�HÐ
|
|||
|
|
,¨hÔ7Ü
|
|||
|
|
�-Š-˜Ó
|
|||
|
|
!€C܇H‚HÐ
|
|||
|
|
.Ð;°·±Ô;ä �”ZÓ €BÜ ˜b �jÓ)€B€v�Jä�2‹w€HØ �‰˜6˜(ˆÓ #× (Ñ (Ó *€BÜ
|
|||
|
|
ˆ2ƒw�ÔÜ � ‰ Ð9¸8ÄcÈ"ÃgÕ;MÔNð ˆ=Õß ‰Ô
|
|||
|
|
Ó ß ‰�—‘�}¤b§i¢i°eÔ&<Ó=Ó >ð €Ñð ˜&•z—}‘}×)Ñ)€B€v�Jؘ&•z—}‘}×*Ñ*€B€w�Kؘ&•z—}‘}×.Ñ.¨sÓ3€B€|Ñó GˆØ —*‘*Ö Ø•g—n‘n¤SÓ)×-Ñ-×3Ñ3Ó5ˆB‰GØ•g—o‘o u¬b¯e©e°V¼R¿U¹UÀBÌÏÉÐ&NÓOˆB‹Gñ Gó PˆØ —*‘*Ö Ü—m’m B¥G°HÔ=ˆB‹Gñ Pô ‡H�HÐ
|
|||
|
|
1Ü
|
|||
|
|
�‹WØ=LÐPRÔ=RˆR�Õ
|
|||
|
|
×
|
|||
|
|
-Ñ
|
|||
|
|
-Ó
|
|||
|
|
/×
|
|||
|
|
7Ñ
|
|||
|
|
7Ó
|
|||
|
|
9ô]ð €Ið Y\ô]ð €Ir c ó € V ^8„ d QhRRRR/# r] r )r s "r r r ô s € ÷ 0ñ 0˜Cð 0¨|ñ 0r c ó& € \ V 4 pVP 4 ' g \ RV 24 h\ P RV 4 \
|
|||
|
|
P ! V 4 p\ P ! R.VP O5! \ V\ 4 p\ VR, 4 VR& \ V4 pVP R.R7 P 4 p\ V4 V8 d' \ P RV\ V4 ,
|
|||
|
|
4 VR, P P VR& VR, P P" VR& VR, P P% R 4 VR
|
|||
|
|
& R F“ pWSP& 9 g K W5, P) \* 4 P* P- 4 W5& W5, P/ R \
|
|||
|
|
P0 R
|
|||
|
|
\
|
|||
|
|
P0 R\
|
|||
|
|
P0 /4 W5& K• RVP& 9 d# \
|
|||
|
|
P2 ! VR, RR7 VR& \ P R\ V4 R V9 d5 \5 VR , P 4 P7 4 4 4 V# R4 V# )aÕ
|
|||
|
|
Load LLC_Data.xlsx and return a normalised DataFrame.
|
|||
|
|
|
|||
|
|
LLC_Data is a supplementary export of Leader Learning Conversations,
|
|||
|
|
often containing richer free-text fields (topic, at-risk observations,
|
|||
|
|
review & action notes) than the Safety_Energy export.
|
|||
|
|
|
|||
|
|
Normalised columns (see LLC_COL_MAP):
|
|||
|
|
date, topic, leader, business_unit, project, location,
|
|||
|
|
crp_focus, at_risk_obs, positive_obs, at_risk_flag, participants
|
|||
|
|
|
|||
|
|
Also adds:
|
|||
|
|
year, month, year_month (Period[M])
|
|||
|
|
zLLC Data file not found: zLoading LLC Data from %sr` rU ra rc rd re rf rg ri rj rk rl Ú at_risk_flagrŒ r� z! Loaded %d LLC records | BUs: %srn )ri rq r� ÚtopicÚ crp_focus) r rt ru r rv r+ rw rx rG r r> r/ ry rB rz r: rd re r{ r r} r r. r~ r r• r€ r� r˜ s & r Ú
|
|||
|
|
load_llc_datarŸ ô sð € ô �‹>€DØ �;‰;�=Š=Ü Ð";¸H¸:Ð FÓGÐGä‡H�HÐ
|
|||
|
|
'¨Ô2Ü
|
|||
|
|
�-Š-˜Ó
|
|||
|
|
!€C܇H‚HÐ
|