A/B testing - canary with session affinity

[stefanprodan]

Allow A/B testing scenarios where instead of weighted routing the traffic is split between the primary and canary based on HTTP headers or cookies. This is particularly useful for frontend applications where you want to have session affinity.

You can chose from two types of analysis:

Progressive weight

• canaryAnalysis.maxWeight
• canaryAnalysis.stepWeight

Match conditions

• canaryAnalysis.iterations
• canaryAnalysis.match

Example:

apiVersion: flagger.app/v1alpha3
kind: Canary
spec:
  canaryAnalysis:
    # schedule interval (default 60s)
    interval: 1m
    # total number of iterations
    iterations: 10
    # max number of failed iterations before rollback
    threshold: 2
    # canary match condition
    match:
      - headers:
          # target Safari users
          user-agent:
            regex: "^(?!.*Chrome)(?=.*\bSafari\b).*$"
      - headers:
          # target users based on a cookie content
          cookie:
            regex: "^(.*?;)?(user=test)(;.*)?$"

The above configuration will run an analysis for ten minutes targeting Safari users and those that have a test cookie. You can determine the minimum time that it takes to validate and promote a canary deployment using this formula:

interval * iterations

And the time it takes for a canary to be rollback when the metrics or webhook checks are failing:

interval * threshold 

Fix: #87

Test image: quay.io/stefanprodan/flagger:ab-testing-e80a3d3

[codecov-io]

Codecov Report

Merging #88 into master will increase coverage by 2.77%.
The diff coverage is 75.75%.

@@            Coverage Diff             @@
##           master      #88      +/-   ##
==========================================
+ Coverage    54.8%   57.58%   +2.77%     
==========================================
  Files          13       13              
  Lines        1540     1794     +254     
==========================================
+ Hits          844     1033     +189     
- Misses        556      605      +49     
- Partials      140      156      +16

Impacted Files	Coverage Δ
pkg/router/istio.go	87.61% <100%> (+6.52%)	⬆️
pkg/controller/deployer.go	57.92% <44.44%> (-0.79%)	⬇️
pkg/controller/scheduler.go	49.41% <51.61%> (+9.85%)	⬆️
pkg/controller/job.go	0% <0%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fcd5207...12ac96d. Read the comment docs.

[huydinhle]

Don't we have to include match field in the crd as well?

[stefanprodan]

We don't have validation for any of the Istio types (HTTPMatchRequest, HTTPRewrite, HTTPRetry, Headers and CorsPolicy) used in the Canary CRD. This could be addressed in a separate PR.

[huydinhle]

sounds good

[huydinhle]

👍