Data Integration Tool: Talend
- Rick recommended ELT process, however, Steve's preference is to use ETL process.- DARTS Consultant Outcomes
- Training: https://www.talend.com/services/training/catalog/
- Can we set up a mentor/consultant relationship as a guide to follow up after initial training?
- Applicant Extract- get final acceptance from admissions to take the 5-year window off
Database/ Server: MS SQL Server
- Some SQL codes are different from Oracle
- Don't want to re-create the whole table or view. The new table/ view should be smaller and focused.
- Re-design for Finance Tables? - FIN Calculated Fields & Expressions
- Re-design for security roles? and documentation
Business Intelligence Tool: MS Power BI
set up security on table base? dashboard base? (can we set up private tab on the dashboard?)
PowerBI Pro or Premium
Create source tables (eg. degree/ program, dates) to be able to run dashboard better?
Documentation Tool: Data Cookbook
- need more people from Registrar Office?
- Documentation needs to be made a priority in all departments.
- Who will be ultimately responsible for entering the information? Who will be verifying the entered information? Most importantly, who will be ensuring that the verification first, is completed, and second, is completed in a timely fashion?
- Another challenge is ensuring that the technical data AND the more basic end-user definitions are both equally fleshed out. There needs to be a bridge over the gap between the deep data users and the casual end-users; a need to speak both languages.
- It would be important that this tool not be relied on perpetually to show users how to navigate our messy data.
Recommendations from Athena Warehouse Assessment:
Contents of Effective Warehouse Strategy from Rick on October 11th, 2017 as follows:
• Database for DW (staging, integration, distribution & BI (if relational) schemas)
o If Oracle what EE features are NOT supported on your current license?
o If Oracle, can we assume Oracle 12c?
o If Oracle, do you have the Database In-Memory option?
o Would like to leverage in-memory column store, partitioning, query caches, in-memory aggregation in EE with In-memory option….
o If above costs more, how much more? Potentially could look at SQL Server 2017 Standard or Enterprise Editions, with Standard potentially being sufficient.
• Database storage
o Do not know your requirements for V1, V2 or afterwards.
o Guess 2- TBs for both Dev & Prod environments
• Memory on server for DW
o Do not know your requirements for V1.
o Assume database & ELT such as Talend or ODI co-located
o 64Gb minimum. Could be higher if requirement more than I would guess. Would NOT go lower.
• CPUs/Cores for DW server
o Same qualifications (no requirements) and assumptions (dbms & ELT)
o Quad-core CPU. Could be higher if requirement more than I would guess. Would NOT go lower.
o Data Integration Server if not ELT but rather an ETL architecture. This would be in addition to dbms DW server. ETL archietcture is NOT the recommendation.
o Same qualifications (no requirements).
o Will require repository dbms
o 64Gb memory. Could be higher if requirements more than expected or if legacy ETL tool
o 1 TB storage. Could be higher if requirements more than expected or if legacy ETL tool
o Quad-core CPU. Could be higher if requirements more than expected or if legacy ETL tool.