Modern AI systems require more than simple distance checks.
An advanced sensory system evaluates visibility, audibility, threat level, emotional memory, and contextual importance.

This guide implements a layered sensory architecture suitable for stealth, tactical, or systemic RPG games.

1. Architecture Overview

The sensory system is divided into four modules:

1. Visual Sensor (FOV + occlusion + confidence)
2. Audio Sensor (sound events + propagation)
3. Threat Evaluator (danger scoring)
4. Interest Model (non-hostile stimuli)

All feed into a unified Perception Memory System.

Stimulus → Sensor → Scoring → Memory → Behavior System

2. Stimulus Base Class

public enum StimulusType
{
    Visual,
    Sound,
    Damage,
    Suspicious
}

public class Stimulus
{
    public StimulusType type;
    public Transform source;
    public Vector3 position;
    public float intensity;
    public float timestamp;
}

3. Visual Sensor (Field of View + Occlusion)

public class VisualSensor : MonoBehaviour
{
    public float viewDistance = 20f;
    public float viewAngle = 120f;
    public LayerMask obstructionMask;

    public float EvaluateVisibility(Transform target)
    {
        Vector3 dir = (target.position - transform.position);
        float distance = dir.magnitude;

        if (distance > viewDistance) return 0f;

        float angle = Vector3.Angle(transform.forward, dir);
        if (angle > viewAngle * 0.5f) return 0f;

        // Line of sight check
        if (Physics.Raycast(transform.position, dir.normalized, out RaycastHit hit, viewDistance, obstructionMask))
        {
            if (hit.transform != target)
                return 0f;
        }

        // Visibility confidence (distance falloff)
        float visibility = 1f - (distance / viewDistance);
        return Mathf.Clamp01(visibility);
    }
}

✅ Improvements you can add:
– Head bone position instead of root
– Partial body visibility checks
– Dynamic FOV (alert state widens FOV)

4. Audio Sensor (Sound Propagation)

Sound is event-based.

public class SoundEmitter : MonoBehaviour
{
    public static System.Action<Stimulus> OnSound;

    public static void Emit(Vector3 pos, float intensity, Transform source)
    {
        OnSound?.Invoke(new Stimulus
        {
            type = StimulusType.Sound,
            position = pos,
            intensity = intensity,
            source = source,
            timestamp = Time.time
        });
    }
}

AI Listener:

public class AudioSensor : MonoBehaviour
{
    public float hearingRadius = 25f;

    private void OnEnable()
    {
        SoundEmitter.OnSound += ProcessSound;
    }

    private void OnDisable()
    {
        SoundEmitter.OnSound -= ProcessSound;
    }

    void ProcessSound(Stimulus s)
    {
        float dist = Vector3.Distance(transform.position, s.position);
        if (dist > hearingRadius) return;

        float perceivedIntensity = s.intensity / dist;
        if (perceivedIntensity > 0.1f)
        {
            PerceptionMemory.Instance.RegisterStimulus(this, s, perceivedIntensity);
        }
    }
}

✅ Advanced version:
– Raycast for sound obstruction
– Different surfaces absorb sound differently
– AI direction estimation (not exact position)

5. Threat Evaluation System

Not everything seen or heard is a threat.

public class ThreatEvaluator
{
    public static float EvaluateThreat(Stimulus s, float confidence)
    {
        float threat = 0f;

        switch (s.type)
        {
            case StimulusType.Visual:
                threat = confidence * 1.2f;
                break;

            case StimulusType.Sound:
                threat = confidence * 0.6f;
                break;

            case StimulusType.Damage:
                threat = 2f;
                break;
        }

        return threat;
    }
}

Threat score can also include:
– Player weapon drawn?
– AI health low?
– Allies nearby?
– Previous hostility?

This is where systemic depth explodes.

6. Interest System (Curiosity Model)

Not every stimulus leads to combat.
Interest system handles:

• Strange sounds
• Unusual objects
• Dead bodies
• Light changes

public class InterestEvaluator
{
    public static float EvaluateInterest(Stimulus s)
    {
        if (s.type == StimulusType.Sound)
            return s.intensity * 0.8f;

        if (s.type == StimulusType.Suspicious)
            return 1.0f;

        return 0f;
    }
}

This enables:
– Investigate state
– Suspicion buildup
– Patrol deviations

7. Perception Memory System (Core Brain)

public class PerceivedEntity
{
    public Transform target;
    public float threatScore;
    public float interestScore;
    public float lastSeenTime;
}

public class PerceptionMemory : MonoBehaviour
{
    public static PerceptionMemory Instance;

    private Dictionary<Transform, PerceivedEntity> memory = new();

    void Awake()
    {
        Instance = this;
    }

    public void RegisterStimulus(MonoBehaviour owner, Stimulus s, float confidence)
    {
        float threat = ThreatEvaluator.EvaluateThreat(s, confidence);
        float interest = InterestEvaluator.EvaluateInterest(s);

        if (!memory.ContainsKey(s.source))
            memory[s.source] = new PerceivedEntity { target = s.source };

        var entry = memory[s.source];
        entry.threatScore += threat;
        entry.interestScore += interest;
        entry.lastSeenTime = Time.time;
    }

    void Update()
    {
        foreach (var e in memory.Values)
        {
            // decay over time
            e.threatScore *= 0.98f;
            e.interestScore *= 0.97f;
        }
    }
}

✅ This gives:
– Suspicion buildup
– Forgetting over time
– Gradual awareness increase

8. Behavior Integration

Behavior Tree Example:

If ThreatScore > 1.5 → Attack
Else If InterestScore > 0.7 → Investigate
Else → Patrol

Or State Machine:

Idle → Suspicious → Investigating → Alerted → Combat

9. Advanced Extensions (AAA Tier)

• Peripheral Vision falloff curve
• Head tracking during investigation
• Group communication (shared perception memory)
• Emotional state affecting detection thresholds
• Stealth modifiers (light level, crouching, noise profile)
• Memory tagging (enemy, neutral, ally)
• Last Known Position tracking
• Heat maps for repeated disturbances

10. Summary

A high‑complexity sensory system transforms AI from reactive scripts into systemic agents.

• Multi‑layer sensing
• Threat scoring
• Interest modeling
• Memory decay
• Behavior integration

This is the foundation of stealth games, tactical shooters, survival horror, and advanced RPG AI.