---
# === IDENTITY ===
id: consulting/recipes/oia-data-collection/2026
canonical_question: "How do you gather data for an OIA from Slack, email, Jira, Git, calendar, and HRIS?"
aliases:
  - "OIA data gathering process"
  - "Organizational network analysis data pipeline"
  - "How to collect communication metadata for organizational audit"
entity_type: execution_recipe
domain: consulting > recipes > OIA Data Collection
region: global
jurisdiction: global
temporal_scope: 2026-2027

# === VERIFICATION ===
last_verified: 2026-03-29
confidence: 0.85
version: 1.0
first_published: 2026-03-29

# === TEMPORAL VALIDITY ===
temporal_validity:
  status: evolving
  last_breaking_change: "Initial release — data collection protocols v1.0"
  next_review: 2026-09-25
  change_sensitivity: high

# === CONSTRAINTS ===
constraints:
  - "GDPR: use metadata only — no message content without explicit consent"
  - "Data processing agreement must be signed before any API access"
  - "Minimum 90 days of historical data required for meaningful network analysis"
  - "All individual-level data must be anonymized before analysis"
  - "Data retention: engagement duration + 30 days, then delete"

# === SKIP CONDITIONS ===
skip_this_unit_if:
  - condition: "User needs full OIA engagement plan, not just data collection"
    use_instead: "consulting/recipes/oia-engagement-playbook/2026"
  - condition: "User already has network data and needs analysis"
    use_instead: "consulting/recipes/oia-network-analysis-execution/2026"
  - condition: "User needs survey design only, not system data"
    use_instead: "consulting/recipes/oia-autoimmune-scan-execution/2026"

# === AGENT HINTS ===
inputs_needed:
  - key: platforms
    question: "Which communication platforms does the organization use?"
    type: choice
    options: ["Slack + Google Workspace", "Slack + Microsoft 365", "Microsoft Teams + Microsoft 365", "mixed/other"]
  - key: admin_access
    question: "Do you have admin-level API access to these platforms?"
    type: choice
    options: ["yes — full admin", "partial — some platforms", "no — need to negotiate", "unknown"]
  - key: privacy_jurisdiction
    question: "Which privacy jurisdiction applies?"
    type: choice
    options: ["GDPR (EU)", "CCPA (California)", "no specific regulation", "multiple jurisdictions"]
  - key: historical_depth
    question: "How many months of historical data are available?"
    type: choice
    options: ["3 months", "6 months", "12+ months", "unknown"]

# === EXECUTION METADATA ===
execution:
  required_inputs:
    - name: "Signed scope document"
      source: "consulting/recipes/oia-engagement-playbook/2026"
      format: "document"
    - name: "IT system inventory"
      source: "client/IT-department"
      format: "spreadsheet"
    - name: "Data processing agreement"
      source: "client/legal"
      format: "document"

  outputs:
    - name: "Communication metadata dataset"
      format: "CSV + JSON"
      description: "Anonymized sender/receiver pairs with timestamps, channel names, interaction types across all platforms"
    - name: "Project tool dataset"
      format: "CSV"
      description: "Ticket velocity, assignment patterns, PR review turnaround, meeting density"
    - name: "Org structure dataset"
      format: "JSON"
      description: "Org chart with tenure, role criticality, department mapping"

  tools_required:
    - name: "Slack API (slack_sdk)"
      purpose: "Extract communication metadata from Slack"
      tier: "paid"
      cost: "$0 (included in Slack Business+)"
      alternatives: ["Slack Admin Export (manual)"]
    - name: "Microsoft Graph API"
      purpose: "Extract email metadata and calendar data"
      tier: "paid"
      cost: "$0 (included in Microsoft 365)"
      alternatives: ["Google Workspace Admin SDK"]
    - name: "Jira REST API"
      purpose: "Extract project management data"
      tier: "free"
      cost: "$0"
      alternatives: ["Asana API", "Linear API", "GitHub Projects API"]
    - name: "Python 3.10+"
      purpose: "Data processing and anonymization scripts"
      tier: "free"
      cost: "$0"
      alternatives: []

  credentials_needed:
    - service: "Slack"
      type: "Bot OAuth token (xoxb-)"
      where_to_get: "https://api.slack.com/apps"
      free_tier_limits: "N/A — requires Slack Business+ for full export"
    - service: "Microsoft Graph"
      type: "OAuth 2.0 token with Mail.Read and Calendars.Read"
      where_to_get: "https://portal.azure.com"
      free_tier_limits: "N/A — requires admin consent"
    - service: "Jira"
      type: "API token"
      where_to_get: "https://id.atlassian.com/manage-profile/security/api-tokens"
      free_tier_limits: "No limit on reads for cloud instances"

  estimated_duration: "2-5 days"
  estimated_cost: "$0 (standard admin tools) to $500 (API costs at scale)"

# === DISTRIBUTION ===
canonical_source: "https://knowledgelib.io/consulting/recipes/oia-data-collection/2026"
suggested_citation: "Source: knowledgelib.io — AI Knowledge Library (verified 2026-03-29)"

# === RELATED UNITS ===
related_kos:
  depends_on:
    - id: "consulting/recipes/oia-engagement-playbook/2026"
      label: "Master engagement playbook this sub-recipe serves"
  feeds_into:
    - id: "consulting/recipes/oia-network-analysis-execution/2026"
      label: "Network analysis recipe that consumes this data"
  related_to:
    - id: "consulting/oia/ambient-exhaust-monitoring/2026"
      label: "Ambient exhaust monitoring theory"

# === SOURCES ===
sources:
  - id: src1
    title: "Slack API Documentation"
    author: Slack
    url: https://api.slack.com/
    type: official_docs
    published: 2026-01-01
    reliability: authoritative
  - id: src2
    title: "Microsoft Graph API Documentation"
    author: Microsoft
    url: https://learn.microsoft.com/en-us/graph/
    type: official_docs
    published: 2026-01-01
    reliability: authoritative
  - id: src3
    title: "GDPR Recital 47 — Legitimate interest basis for metadata processing"
    author: European Union
    url: https://gdpr-info.eu/recitals/no-47/
    type: official_docs
    published: 2018-05-25
    reliability: authoritative
  - id: src4
    title: "The New Science of Building Great Teams"
    author: Pentland, A.
    url: https://hbr.org/2012/04/the-new-science-of-building-great-teams
    type: primary_research
    published: 2012-04-01
    reliability: authoritative
---

# OIA Data Collection

## Purpose

This recipe produces an anonymized, multi-platform communication metadata dataset ready for organizational network analysis. It extracts interaction patterns from Slack, email, Jira, Git, calendar, and HRIS — capturing who communicates with whom, how often, and through which channels — without accessing message content. The output feeds directly into the network analysis execution recipe. [src3, src4]

## Prerequisites
<!-- Agents: verify ALL prerequisites before executing. Missing prerequisites = failed execution. -->

- [ ] **Signed scope document** from [OIA Engagement Playbook](/consulting/recipes/oia-engagement-playbook/2026)
- [ ] **Data processing agreement** — signed by client legal, specifying metadata-only access
- [ ] **Slack Bot OAuth token** — from [Slack API](https://api.slack.com/apps) (requires Business+ plan)
- [ ] **Microsoft Graph OAuth token** — from [Azure Portal](https://portal.azure.com) (requires admin consent)
- [ ] **Jira API token** — from [Atlassian](https://id.atlassian.com/manage-profile/security/api-tokens)
- [ ] **Python 3.10+** with pip installed

## Constraints
<!-- Hard rules. Agents: enforce throughout execution. Violating these = broken output or legal risk. -->

- GDPR: extract metadata only — timestamps, sender/receiver IDs, channel names, reaction counts. Never message body content. [src3]
- All individual identifiers must be pseudonymized (hash employee IDs) before any analysis.
- Minimum 90 days of historical data required. Less than 90 days produces seasonal bias. [src4]
- Data retention: delete all raw exports within 30 days of engagement completion.
- Rate limits: Slack API (20 req/min for conversations.history), Microsoft Graph (10,000 req/10min). Pace accordingly. [src1, src2]

## Tool Selection Decision

```
Which platform combination?
├── Slack + Google Workspace
│   └── PATH A: Slack API + Google Admin SDK
├── Slack + Microsoft 365
│   └── PATH B: Slack API + Microsoft Graph API
├── Microsoft Teams + Microsoft 365
│   └── PATH C: Microsoft Graph API (unified)
└── Mixed or restricted access
    └── PATH D: Manual exports + survey supplement
```

| Path | Tools | Cost | Speed | Output Quality |
|------|-------|------|-------|---------------|
| A: Slack + Google | slack_sdk, Google Admin SDK, Python | $0 | 2-3 days | Excellent — full API coverage |
| B: Slack + Microsoft | slack_sdk, Microsoft Graph, Python | $0 | 2-3 days | Excellent — full API coverage |
| C: Teams + Microsoft | Microsoft Graph API, Python | $0 | 1-2 days | Excellent — unified platform |
| D: Manual + Survey | Admin exports, survey tool | $0-$200 | 3-5 days | Adequate — gaps in coverage |

## Execution Flow

### Step 1: Inventory Available Systems

**Duration**: 2-4 hours
**Tool**: Client IT coordination

Map all communication and collaboration platforms in use across in-scope departments. For each system, document:
- Platform name and version
- API availability and admin access status
- Data retention period (how far back data goes)
- User count in scope

```
System Inventory Template:
| Platform | Type | API Available | Admin Access | Data Depth | Users |
|----------|------|--------------|-------------|------------|-------|
| Slack    | Chat | Yes (Business+) | Confirmed | 12 months | 450 |
| M365     | Email | Yes (Graph) | Pending | 24 months | 500 |
| Jira     | PM   | Yes (REST) | Confirmed | 18 months | 200 |
```

**Verify**: Complete system list with API availability confirmed for each platform.
**If failed**: If admin access unavailable for a system, document the gap and proceed with available systems. Supplement with survey data.

### Step 2: Legal & Privacy Framework

**Duration**: 1-2 days
**Tool**: Legal coordination + document templates

Draft and execute privacy compliance framework:
- Data processing agreement specifying metadata-only scope
- Employee communication (transparency notice — not consent request for metadata under legitimate interest) [src3]
- Anonymization protocol: SHA-256 hash of employee ID + engagement salt
- Data retention schedule: raw data deleted engagement + 30 days

```python
# Anonymization function — apply to all employee identifiers
import hashlib

ENGAGEMENT_SALT = "oia-{client_id}-{engagement_date}"

def anonymize_id(employee_id: str) -> str:
    """Pseudonymize employee ID using salted hash."""
    return hashlib.sha256(
        f"{employee_id}{ENGAGEMENT_SALT}".encode()
    ).hexdigest()[:12]
```

**Verify**: Legal sign-off received, employee transparency notice sent.
**If failed**: If legal blocks metadata access, escalate to executive sponsor. If still blocked, fall back to PATH D (manual exports + survey).

### Step 3: Slack Data Export

**Duration**: 4-8 hours
**Tool**: Python slack_sdk or Slack Admin Panel

Extract communication metadata from Slack. Target data: message timestamps, channel names, reaction counts, thread depths, sender IDs (anonymized). NOT message content. [src1]

```python
# Slack metadata extraction
from slack_sdk import WebClient
import pandas as pd

client = WebClient(token="xoxb-YOUR-BOT-TOKEN")

# List all channels in scope
channels = []
cursor = None
while True:
    resp = client.conversations_list(cursor=cursor, limit=200, types="public_channel,private_channel")
    channels.extend(resp["channels"])
    cursor = resp.get("response_metadata", {}).get("next_cursor")
    if not cursor:
        break

# Extract metadata per channel (no message content)
metadata_rows = []
for ch in channels:
    history = client.conversations_history(channel=ch["id"], limit=1000)
    for msg in history["messages"]:
        metadata_rows.append({
            "channel": ch["name"],
            "sender_id": anonymize_id(msg.get("user", "unknown")),
            "timestamp": msg["ts"],
            "has_thread": "thread_ts" in msg,
            "reply_count": msg.get("reply_count", 0),
            "reaction_count": len(msg.get("reactions", [])),
        })

df_slack = pd.DataFrame(metadata_rows)
df_slack.to_csv("slack_metadata.csv", index=False)
```

**Verify**: Export covers all target departments. Row count > 1000 for a typical 90-day window.
**If failed**: If API rate limited, reduce batch size and add sleep intervals (1s between requests). If admin export is faster, use Slack Admin Panel export and filter programmatically.

### Step 4: Email Metadata Collection

**Duration**: 4-8 hours
**Tool**: Microsoft Graph API or Google Workspace Admin SDK

Extract email metadata: sender/receiver pairs, timestamps, volume per relationship. NOT body content. [src2]

```python
# Microsoft Graph email metadata extraction
import requests

headers = {"Authorization": f"Bearer {access_token}"}

# For each user in scope, extract sent mail metadata
for user_id in in_scope_users:
    url = f"https://graph.microsoft.com/v1.0/users/{user_id}/messages"
    params = {
        "$select": "sender,toRecipients,ccRecipients,sentDateTime",
        "$top": 1000,
        "$filter": f"sentDateTime ge {start_date}"
    }
    resp = requests.get(url, headers=headers, params=params)
    messages = resp.json().get("value", [])

    for msg in messages:
        metadata_rows.append({
            "sender_id": anonymize_id(user_id),
            "recipients": [anonymize_id(r["emailAddress"]["address"])
                          for r in msg.get("toRecipients", [])],
            "cc_count": len(msg.get("ccRecipients", [])),
            "timestamp": msg["sentDateTime"],
        })
```

**Verify**: Metadata covers engagement timeframe (90+ days). Cross-reference user count with org chart.
**If failed**: If Microsoft Graph throttles (429), implement exponential backoff. If Google Workspace, use Admin SDK Reports API for aggregate data.

### Step 5: Project Tool Data

**Duration**: 2-4 hours
**Tool**: Jira REST API, GitHub API, calendar API

Extract project management and collaboration metadata:
- Jira: ticket creation/resolution velocity, assignee patterns, comment frequency
- GitHub/GitLab: PR review turnaround, reviewer assignment patterns, commit frequency
- Calendar: meeting density, cross-team meeting ratio, 1:1 frequency

```python
# Jira metadata extraction
import requests

jira_url = "https://your-domain.atlassian.net"
headers = {"Authorization": f"Basic {api_token_b64}"}

# Search for issues in scope
jql = f"project in ({projects}) AND created >= -{days_back}d"
resp = requests.get(f"{jira_url}/rest/api/3/search",
    headers=headers,
    params={"jql": jql, "maxResults": 1000,
            "fields": "assignee,reporter,created,resolutiondate,status"})

issues = resp.json()["issues"]
for issue in issues:
    fields = issue["fields"]
    metadata_rows.append({
        "assignee_id": anonymize_id(fields["assignee"]["accountId"]),
        "reporter_id": anonymize_id(fields["reporter"]["accountId"]),
        "created": fields["created"],
        "resolved": fields.get("resolutiondate"),
        "status": fields["status"]["name"],
    })
```

**Verify**: Data covers 90+ days. Ticket count > 100 for meaningful velocity analysis.
**If failed**: If Jira API unavailable, use CSV export from Jira admin. If GitHub, use `gh api` CLI.

### Step 6: HRIS Data Integration

**Duration**: 2-4 hours
**Tool**: HRIS export (BambooHR, Workday, or manual)

Extract organizational structure data: reporting lines, department assignments, tenure, role titles, location. Cross-reference with communication data to validate network against formal structure.

```python
# HRIS data processing
import pandas as pd

# Load HRIS export (typically CSV from admin panel)
df_hris = pd.read_csv("hris_export.csv")

# Anonymize and structure
df_org = pd.DataFrame({
    "employee_id": df_hris["employee_id"].apply(anonymize_id),
    "manager_id": df_hris["manager_id"].apply(anonymize_id),
    "department": df_hris["department"],
    "tenure_months": df_hris["tenure_months"],
    "role_level": df_hris["role_level"],
    "location": df_hris["location"],
})

# Merge with communication data
df_combined = df_slack.merge(df_org, left_on="sender_id", right_on="employee_id", how="left")
df_combined.to_csv("combined_oia_dataset.csv", index=False)
```

**Verify**: Cross-referenced with communication data — > 90% of communication senders matched to HRIS records.
**If failed**: If HRIS export unavailable, build org chart manually from client-provided document. This adds 1-2 days.

## Output Schema

```json
{
  "output_type": "oia_communication_dataset",
  "format": "CSV + JSON",
  "files": [
    {"name": "slack_metadata.csv", "description": "Slack channel activity metadata"},
    {"name": "email_metadata.csv", "description": "Email sender/receiver pair metadata"},
    {"name": "project_metadata.csv", "description": "Jira/GitHub project management metadata"},
    {"name": "org_structure.json", "description": "Anonymized org chart with department mapping"},
    {"name": "combined_oia_dataset.csv", "description": "Merged dataset ready for network analysis"}
  ],
  "expected_row_count": "5000-50000 (depends on org size and timeframe)",
  "deduplication_key": "sender_id + timestamp + channel"
}
```

## Quality Benchmarks

| Quality Metric | Minimum Acceptable | Good | Excellent |
|---------------|-------------------|------|-----------|
| Employee coverage (% matched to HRIS) | > 70% | > 85% | > 95% |
| Temporal coverage (days of data) | > 90 days | > 180 days | > 365 days |
| Platform coverage (% of systems integrated) | > 50% | > 75% | > 90% |
| Anonymization completeness | 100% | 100% | 100% |
| Data freshness (days since last record) | < 30 days | < 14 days | < 7 days |

**If below minimum**: Extend data collection period, add missing platform integrations, or supplement with survey data.

## Error Handling

| Error | Likely Cause | Recovery Action |
|-------|-------------|----------------|
| Slack API rate limit (429) | Too many requests per minute | Add 2s sleep between requests, reduce batch size to 50 |
| Microsoft Graph auth failure (401) | Token expired or insufficient permissions | Refresh OAuth token, verify Mail.Read and Calendars.Read scopes |
| HRIS export incomplete | Terminated employees excluded or department filter applied | Request full export including all active employees in scope |
| Low employee match rate (< 70%) | Email addresses don't match Slack usernames | Build manual mapping table from HRIS email field to Slack user ID |
| Data gap in timeframe | Platform retention policy shorter than 90 days | Document gap, supplement with survey data for missing period |

## Cost Breakdown

| Component | Free Tier | Paid Tier | At Scale |
|-----------|-----------|-----------|----------|
| Slack API access | $0 (Business+ included) | $0 | $0 |
| Microsoft Graph API | $0 (M365 included) | $0 | $0 |
| Jira API | $0 | $0 | $0 |
| Python + libraries | $0 | $0 | $0 |
| Survey tool (if needed) | $0 (Google Forms) | $50 (Typeform) | $200 (SurveyMonkey) |
| **Total** | **$0** | **$0-$50** | **$0-$500** |

## Anti-Patterns

### Wrong: Accessing message content without consent
Reading actual Slack messages or email bodies to "get richer data." Result: GDPR violation, client trust destroyed, potential legal liability. [src3]

### Correct: Metadata only, always
Extract only metadata: timestamps, sender/receiver pairs, channel names, reaction counts, thread depths. Metadata under legitimate interest basis is legally defensible. Message content requires explicit consent.

### Wrong: Skipping anonymization during collection
Processing data with real names and anonymizing later. Result: raw data with PII exists on consultant's machine, creating data breach risk and GDPR compliance gap. [src3]

### Correct: Anonymize at point of extraction
Apply anonymization (hash) immediately when extracting from each API. Raw employee IDs should never persist in any file on the consultant's systems.

### Wrong: Using only one communication channel
Extracting Slack data alone and declaring network analysis complete. Result: misses email-heavy relationships (common in senior leadership) and project tool interactions, producing skewed network topology. [src4]

### Correct: Multi-channel triangulation
Extract from all available platforms and merge. The combined view reveals relationships invisible in any single channel.

## When This Matters

Use when an agent needs to collect organizational communication metadata for network analysis. This is Step 2-3 of the OIA engagement lifecycle. Requires signed data processing agreement and admin-level API access as prerequisites. Output feeds directly into the network analysis execution recipe.

## Related Units

- [OIA Engagement Playbook](/consulting/recipes/oia-engagement-playbook/2026)
- [OIA Network Analysis Execution](/consulting/recipes/oia-network-analysis-execution/2026)
- [Ambient Exhaust Monitoring Theory](/consulting/oia/ambient-exhaust-monitoring/2026)