Big Data Business Intelligence for Criminal Intelligence Analysis Training Course

Day 01

Introduction to Big Data Business Intelligence in Criminal Intelligence Analysis

• Case Studies in Law Enforcement - Predictive Policing
• Adoption rates of Big Data in law enforcement agencies and strategies for aligning future operations with Big Data predictive analytics
• Emerging technology solutions such as gunshot detection sensors, surveillance video, and social media analytics
• Leveraging Big Data technologies to manage information overload
• Integrating Big Data with legacy data systems
• Foundational understanding of enabling technologies for predictive analytics
• Data integration and dashboard visualization
• Fraud management strategies
• Business rules and fraud detection mechanisms
• Threat detection and profiling
• Cost-benefit analysis for Big Data implementation

Fundamentals of Big Data

• Core characteristics of Big Data: Volume, Variety, Velocity, and Veracity.
• Massively Parallel Processing (MPP) architecture
• Data Warehouses: static schemas and slowly evolving datasets
• MPP Databases: Greenplum, Exadata, Teradata, Netezza, Vertica, among others.
• Hadoop-based solutions: flexible structures without strict dataset constraints.
• Common patterns: HDFS, MapReduce (crunch), and retrieval from HDFS
• Apache Spark for stream processing
• Batch processing: ideal for analytical and non-interactive tasks
• Volume: CEP (Complex Event Processing) streaming data
• Common choices: CEP products (e.g., Infostreams, Apama, MarkLogic)
• Less production-ready options: Storm/S4
• NoSQL Databases: columnar and key-value stores, best used as analytical adjuncts to data warehouses

NoSQL Solutions

• KV Store: Keyspace, Flare, SchemaFree, RAMCloud, Oracle NoSQL Database (OnDB)
• KV Store: Dynamo, Voldemort, Dynomite, SubRecord, Mo8onDb, DovetailDB
• KV Store (Hierarchical): GT.m, Cache
• KV Store (Ordered): TokyoTyrant, Lightcloud, NMDB, Luxio, MemcacheDB, Actord
• KV Cache: Memcached, Repcached, Coherence, Infinispan, EXtremeScale, JBossCache, Velocity, Terracoqua
• Tuple Store: Gigaspaces, Coord, Apache River
• Object Database: ZopeDB, DB40, Shoal
• Document Store: CouchDB, Cloudant, Couchbase, MongoDB, Jackrabbit, XML-Databases, ThruDB, CloudKit, Prsevere, Riak-Basho, Scalaris
• Wide Columnar Store: BigTable, HBase, Apache Cassandra, Hypertable, KAI, OpenNeptune, Qbase, KDI

Data Varieties: Introduction to Data Cleaning Challenges in Big Data

• RDBMS: static structure/schema, which limits agile and exploratory environments.
• NoSQL: semi-structured data, offering sufficient structure for storage without a rigid pre-defined schema
• Key data cleaning issues

Hadoop

• Criteria for selecting Hadoop
• Structured Data: Enterprise data warehouses/databases can handle massive volumes (at a cost) but enforce structure, hindering active exploration
• Semi-Structured Data: challenging to manage with traditional solutions (DW/DB)
• Data warehousing involves significant effort and remains static post-implementation
• Handling variety and volume on commodity hardware: Hadoop
• Commodity hardware requirements for establishing a Hadoop cluster

Introduction to MapReduce / HDFS

• MapReduce: distributing computational tasks across multiple servers
• HDFS: ensuring local data availability for processing with redundancy
• Data handling: supports unstructured/schema-less data (unlike RDBMS)
• Developer responsibility for interpreting data semantics
• Programming MapReduce involves working with Java (advantages/disadvantages) and manually loading data into HDFS

Day 02

The Big Data Ecosystem -- Building Big Data ETL (Extract, Transform, Load) -- Selecting Appropriate Big Data Tools

• Comparing Hadoop with other NoSQL solutions
• Requirements for interactive, random access to data
• HBase (column-oriented database) integrated with Hadoop
• Random access capabilities with constraints (max 1 PB)
• Limitations for ad-hoc analytics; suitability for logging, counting, and time-series data
• Sqoop: importing data from databases to Hive or HDFS (JDBC/ODBC access)
• Flume: streaming data (e.g., log data) into HDFS

Big Data Management Systems

• Managing component lifecycle, compute node startup/failure: ZooKeeper for configuration, coordination, and naming services
• Complex pipelines/workflows: Oozie for managing workflows, dependencies, and chaining
• Deployment, configuration, cluster management, and upgrades (sysadmin): Ambari
• Cloud deployment: Whirr

Predictive Analytics -- Core Techniques and Machine Learning-Based Business Intelligence

• Introduction to Machine Learning
• Learning classification techniques
• Bayesian Prediction: preparing training files
• Support Vector Machines
• KNN p-Tree Algebra & vertical mining
• Neural Networks
• Addressing large variable problems in Big Data: Random Forest (RF)
• Big Data automation challenges: Multi-model ensemble RF
• Automation via Soft10-M
• Text analysis tool: Treeminer
• Agile learning methodologies
• Agent-based learning
• Distributed learning
• Introduction to open-source tools for predictive analytics: R, Python, Rapidminer, Mahut

The Predictive Analytics Ecosystem and Its Application in Criminal Intelligence Analysis

• Technology and the investigative process
• Insight analytics
• Visualization analytics
• Structured predictive analytics
• Unstructured predictive analytics
• Threat, fraudster, and vendor profiling
• Recommendation engines
• Pattern detection
• Rule and scenario discovery: failures, fraud, and optimization
• Root cause analysis
• Sentiment analysis
• CRM analytics
• Network analytics
• Text analytics for extracting insights from transcripts, witness statements, and online chatter
• Technology-assisted review
• Fraud analytics
• Real-time analytics

Day 03

Real-Time and Scalable Analytics on Hadoop

• Reasons why standard analytical algorithms fail on Hadoop/HDFS
• Apache Hama: for Bulk Synchronous distributed computing
• Apache Spark: for cluster computing and real-time analytics
• CMU Graphics Lab2: Graph-based asynchronous approach to distributed computing
• KNN p: Algebra-based approach from Treeminer to reduce operational hardware costs

Tools for eDiscovery and Forensics

• eDiscovery over Big Data vs. Legacy data – comparing costs and performance
• Predictive coding and Technology Assisted Review (TAR)
• Live demonstration of vMiner to illustrate how TAR accelerates discovery
• Accelerated indexing via HDFS – handling data velocity
• NLP (Natural Language Processing): open-source products and techniques
• eDiscovery in foreign languages: technologies for multilingual processing

Big Data BI for Cyber Security – Achieving a 360-Degree View, Rapid Data Collection, and Threat Identification

• Foundations of security analytics: attack surface, misconfigurations, and host defenses
• Network infrastructure, large data pipelines, and response ETL for real-time analytics
• Prescriptive vs. predictive: Fixed rule-based systems vs. automatic threat rule discovery from metadata

Aggregating Disparate Data for Criminal Intelligence Analysis

• Utilizing IoT (Internet of Things) as sensors for data capture
• Using satellite imagery for domestic surveillance
• Leveraging surveillance and image data for criminal identification
• Other data acquisition technologies: drones, body cameras, GPS tagging, and thermal imaging
• Combining automated data retrieval with information from informants, interrogations, and research
• Forecasting criminal activity

Day 04

Fraud Prevention BI from Big Data in Fraud Analytics

• Basic classification of Fraud Analytics: rules-based vs. predictive analytics
• Supervised vs. unsupervised machine learning for fraud pattern detection
• Types of fraud: B2B, medical claims, insurance, tax evasion, and money laundering

Social Media Analytics -- Intelligence Gathering and Analysis

• How criminals use social media for organization, recruitment, and planning
• Big Data ETL APIs for extracting social media data
• Analyzing text, images, metadata, and video
• Sentiment analysis from social media feeds
• Contextual and non-contextual filtering of social media feeds
• Social Media Dashboards for integrating diverse platforms
• Automated profiling of social media accounts
• Live demos of each analytical capability using the Treeminer Tool

Big Data Analytics in Image Processing and Video Feeds

• Image storage techniques in Big Data: solutions for data exceeding petabytes
• LTFS (Linear Tape File System) and LTO (Linear Tape Open)
• GPFS-LTFS (General Parallel File System - Linear Tape File System): layered storage for large image data
• Fundamentals of image analytics
• Object recognition
• Image segmentation
• Motion tracking
• 3-D image reconstruction

Biometrics, DNA, and Next-Generation Identification Programs

• Advancements beyond fingerprinting and facial recognition
• Speech recognition, keystroke dynamics, and CODIS (Combined DNA Index System)
• Going beyond DNA matching: using forensic DNA phenotyping to construct facial profiles from DNA samples

Big Data Dashboards for Quick Accessibility and Display of Diverse Data

• Integrating existing application platforms with Big Data Dashboards
• Big Data management
• Case study of Big Data Dashboards: Tableau and Pentaho
• Using Big Data applications to enable location-based services in government
• Tracking systems and management

Day 05

Justifying Big Data BI Implementation Within an Organization

• Defining ROI (Return on Investment) for Big Data implementation
• Case studies on saving analyst time in data collection and preparation – boosting productivity
• Revenue gains from reduced database licensing costs
• Revenue gains from location-based services
• Cost savings through fraud prevention
• An integrated spreadsheet approach for estimating costs versus revenue gains/savings from Big Data implementation

Step-by-Step Procedure for Replacing Legacy Data Systems with Big Data Systems

• Big Data Migration Roadmap
• Critical information required before architecting a Big Data system
• Methods for calculating the Volume, Velocity, Variety, and Veracity of data
• Estimating data growth
• Case studies

Review of Big Data Vendors and Their Products

• Accenture
• APTEAN (Formerly CDC Software)
• Cisco Systems
• Cloudera
• Dell
• EMC
• GoodData Corporation
• Guavus
• Hitachi Data Systems
• Hortonworks
• HP
• IBM
• Informatica
• Intel
• Jaspersoft
• Microsoft
• MongoDB (Formerly 10Gen)
• MU Sigma
• Netapp
• Opera Solutions
• Oracle
• Pentaho
• Platfora
• Qliktech
• Quantum
• Rackspace
• Revolution Analytics
• Salesforce
• SAP
• SAS Institute
• Sisense
• Software AG/Terracotta
• Soft10 Automation
• Splunk
• Sqrrl
• Supermicro
• Tableau Software
• Teradata
• Think Big Analytics
• Tidemark Systems
• Treeminer
• VMware (Part of EMC)

Q/A Session