Egress using Wildcard Hosts

The Accessing External Services task and the Configure an Egress Gateway example describe how to configure egress traffic for specific hostnames, like edition.cnn.com. This example shows how to enable egress traffic for a set of hosts in a common domain, for example *.wikipedia.org, instead of configuring each and every host separately.

Background

Suppose you want to enable egress traffic in Istio for the wikipedia.org sites in all languages. Each version of wikipedia.org in a particular language has its own hostname, e.g., en.wikipedia.org and de.wikipedia.org in the English and the German languages, respectively. You want to enable egress traffic by common configuration items for all the Wikipedia sites, without the need to specify every language’s site separately.

Before you begin

  • Install Istio using the demo configuration profile and with the blocking-by-default outbound traffic policy:

    1. $ istioctl install --set profile=demo --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY

    You can run this task on an Istio configuration other than the demo profile as long as you make sure to deploy the Istio egress gateway, enable Envoy’s access logging, and apply the blocking-by-default outbound traffic policy in your installation. You will also need to add the second gateway using your own IstioOperator CR instead of the one shown in setup egress gateway with SNI proxy.

  • Deploy the sleep sample app to use as a test source for sending requests. If you have automatic sidecar injection enabled, run the following command to deploy the sample app:

    Zip

    1. $ kubectl apply -f @samples/sleep/sleep.yaml@

    Otherwise, manually inject the sidecar before deploying the sleep application with the following command:

    Zip

    1. $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@)

    You can use any pod with curl installed as a test source.

  • Set the SOURCE_POD environment variable to the name of your source pod:

    1. $ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})

Configure direct traffic to a wildcard host

The first, and simplest, way to access a set of hosts within a common domain is by configuring a simple ServiceEntry with a wildcard host and calling the services directly from the sidecar. When calling services directly (i.e., not via an egress gateway), the configuration for a wildcard host is no different than that of any other (e.g., fully qualified) host, only much more convenient when there are many hosts within the common domain.

Note that the configuration below can be easily bypassed by a malicious application. For a secure egress traffic control, direct the traffic through an egress gateway.

Note that the DNS resolution cannot be used for wildcard hosts. This is why the NONE resolution (omitted since it is the default) is used in the service entry below.

  1. Define a ServiceEntry for *.wikipedia.org:

    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: ServiceEntry
    4. metadata:
    5.   name: wikipedia
    6. spec:
    7.   hosts:
    8.   - "*.wikipedia.org"
    9.   ports:
    10.   - number: 443
    11.     name: https
    12.     protocol: HTTPS
    13. EOF

  2. Send HTTPS requests to https://en.wikipedia.org and https://de.wikipedia.org:

    1. $ kubectl exec "$SOURCE_POD" -c sleep -- sh -c 'curl -s https://en.wikipedia.org/wiki/Main_Page | grep -o "<title>.*</title>"; curl -s https://de.wikipedia.org/wiki/Wikipedia:Hauptseite | grep -o "<title>.*</title>"'
    2. <title>Wikipedia, the free encyclopedia</title>
    3. <title>Wikipedia – Die freie Enzyklopädie</title>

Cleanup direct traffic to a wildcard host

  1. $ kubectl delete serviceentry wikipedia

Configure egress gateway traffic to a wildcard host

The configuration for accessing a wildcard host via an egress gateway depends on whether or not the set of wildcard domains are served by a single common host. This is the case for \.wikipedia.org. All of the language-specific sites are served by every one of the wikipedia.org servers. You can route the traffic to an IP of any *.wikipedia.org site, including www.wikipedia.org*, and it will manage to serve any specific site.

In the general case, where all the domain names of a wildcard are not served by a single hosting server, a more complex configuration is required.

Wildcard configuration for a single hosting server

When all wildcard hosts are served by a single server, the configuration for egress gateway-based access to a wildcard host is very similar to that of any host, with one exception: the configured route destination will not be the same as the configured host, i.e., the wildcard. It will instead be configured with the host of the single server for the set of domains.

  1. Create an egress Gateway for \.wikipedia.org*, a destination rule and a virtual service to direct the traffic through the egress gateway and from the egress gateway to the external service.

    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.   name: istio-egressgateway
    6. spec:
    7.   selector:
    8.     istio: egressgateway
    9.   servers:
    10.   - port:
    11.       number: 443
    12.       name: https
    13.       protocol: HTTPS
    14.     hosts:
    15.     - "*.wikipedia.org"
    16.     tls:
    17.       mode: PASSTHROUGH
    18. ---
    19. apiVersion: networking.istio.io/v1alpha3
    20. kind: DestinationRule
    21. metadata:
    22.   name: egressgateway-for-wikipedia
    23. spec:
    24.   host: istio-egressgateway.istio-system.svc.cluster.local
    25.   subsets:
    26.     - name: wikipedia
    27. ---
    28. apiVersion: networking.istio.io/v1alpha3
    29. kind: VirtualService
    30. metadata:
    31.   name: direct-wikipedia-through-egress-gateway
    32. spec:
    33.   hosts:
    34.   - "*.wikipedia.org"
    35.   gateways:
    36.   - mesh
    37.   - istio-egressgateway
    38.   tls:
    39.   - match:
    40.     - gateways:
    41.       - mesh
    42.       port: 443
    43.       sniHosts:
    44.       - "*.wikipedia.org"
    45.     route:
    46.     - destination:
    47.         host: istio-egressgateway.istio-system.svc.cluster.local
    48.         subset: wikipedia
    49.         port:
    50.           number: 443
    51.       weight: 100
    52.   - match:
    53.     - gateways:
    54.       - istio-egressgateway
    55.       port: 443
    56.       sniHosts:
    57.       - "*.wikipedia.org"
    58.     route:
    59.     - destination:
    60.         host: www.wikipedia.org
    61.         port:
    62.           number: 443
    63.       weight: 100
    64. EOF

  2. Create a ServiceEntry for the destination server, www.wikipedia.org.

    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: ServiceEntry
    4. metadata:
    5.   name: www-wikipedia
    6. spec:
    7.   hosts:
    8.   - www.wikipedia.org
    9.   ports:
    10.   - number: 443
    11.     name: https
    12.     protocol: HTTPS
    13.   resolution: DNS
    14. EOF

  3. Send HTTPS requests to https://en.wikipedia.org and https://de.wikipedia.org:

    1. $ kubectl exec "$SOURCE_POD" -c sleep -- sh -c 'curl -s https://en.wikipedia.org/wiki/Main_Page | grep -o "<title>.*</title>"; curl -s https://de.wikipedia.org/wiki/Wikipedia:Hauptseite | grep -o "<title>.*</title>"'
    2. <title>Wikipedia, the free encyclopedia</title>
    3. <title>Wikipedia – Die freie Enzyklopädie</title>

  4. Check the statistics of the egress gateway’s proxy for the counter that corresponds to your requests to \.wikipedia.org*. If Istio is deployed in the istio-system namespace, the command to print the counter is:

    1. $ kubectl exec "$(kubectl get pod -l istio=egressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}')" -c istio-proxy -n istio-system -- pilot-agent request GET clusters | grep '^outbound|443||www.wikipedia.org.*cx_total:'
    2. outbound|443||www.wikipedia.org::208.80.154.224:443::cx_total::2

Cleanup wildcard configuration for a single hosting server

  1. $ kubectl delete serviceentry www-wikipedia
  2. $ kubectl delete gateway istio-egressgateway
  3. $ kubectl delete virtualservice direct-wikipedia-through-egress-gateway
  4. $ kubectl delete destinationrule egressgateway-for-wikipedia

Wildcard configuration for arbitrary domains

The configuration in the previous section worked because all the \.wikipedia.org sites can be served by any one of the wikipedia.org servers. However, this is not always the case. For example, you may want to configure egress control for access to more general wildcard domains like `.comor*.org`.

Configuring traffic to arbitrary wildcard domains introduces a challenge for Istio gateways. In the previous section you directed the traffic to www.wikipedia.org, which was made known to your gateway during configuration. The gateway, however, would not know the IP address of any arbitrary host it receives in a request. This is due to a limitation of Envoy, the proxy used by the default Istio egress gateway. Envoy routes traffic either to predefined hosts, predefined IP addresses, or to the original destination IP address of the request. In the gateway case, the original destination IP of the request is lost since the request is first routed to the egress gateway and its destination IP address is the IP address of the gateway.

Consequently, the Istio gateway based on Envoy cannot route traffic to an arbitrary host that is not preconfigured, and therefore is unable to perform traffic control for arbitrary wildcard domains. To enable such traffic control for HTTPS, and for any TLS, you need to deploy an SNI forward proxy in addition to Envoy. Envoy will route the requests destined for a wildcard domain to the SNI forward proxy, which, in turn, will forward the requests to the destination specified by the SNI value.

The egress gateway with SNI proxy and the related parts of the Istio architecture are shown in the following diagram:

Egress Gateway with SNI proxy

Egress Gateway with SNI proxy

The following sections show you how to redeploy the egress gateway with an SNI proxy and then configure Istio to route HTTPS traffic through the gateway to arbitrary wildcard domains.

Setup egress gateway with SNI proxy

In this section you deploy an egress gateway with an SNI proxy in addition to the standard Istio Envoy proxy. This example uses Nginx for the SNI proxy, although any SNI proxy that is capable of routing traffic according to arbitrary, not-preconfigured, SNI values would do. The SNI proxy will listen on port 8443, although you can use any port other than the ports specified for the egress Gateway and for the VirtualServices bound to it. The SNI proxy will forward the traffic to port 443.

  1. Create a configuration file for the Nginx SNI proxy. You may want to edit the file to specify additional Nginx settings, if required. Note that the listen directive of the server specifies port 8443, its proxy_pass directive uses ssl_preread_server_name with port 443 and ssl_preread is on to enable SNI reading.

    1. $ cat <<EOF > ./sni-proxy.conf
    2. # setup custom path that do not require root access
    3. pid /tmp/nginx.pid;
    5. events {
    6. }
    8. stream {
    9.   log_format log_stream '\$remote_addr [\$time_local] \$protocol [\$ssl_preread_server_name]'
    10.   '\$status \$bytes_sent \$bytes_received \$session_time';
    12.   access_log /var/log/nginx/access.log log_stream;
    13.   error_log  /var/log/nginx/error.log;
    15.   # tcp forward proxy by SNI
    16.   server {
    17.     resolver 8.8.8.8 ipv6=off;
    18.     listen       127.0.0.1:18443;
    19.     proxy_pass   \$ssl_preread_server_name:443;
    20.     ssl_preread  on;
    21.   }
    22. }
    23. EOF

  2. Create a Kubernetes ConfigMap to hold the configuration of the Nginx SNI proxy:

    1. $ kubectl create configmap egress-sni-proxy-configmap -n istio-system --from-file=nginx.conf=./sni-proxy.conf

  3. Create an IstioOperator CR to add a new egress gateway with SNI proxy:

    1. $ istioctl manifest generate -f - <<EOF > ./egressgateway-with-sni-proxy.yaml
    2. apiVersion: install.istio.io/v1alpha1
    3. kind: IstioOperator
    4. spec:
    5.   # Only generate a gateway component defined below.
    6.   # Using this with "istioctl install" will reconcile and remove existing control-plane components.
    7.   # Instead use "istioctl manifest generate" or "kubectl create" if using the istio operator.
    8.   profile: empty
    9.   components:
    10.     egressGateways:
    11.     - name: istio-egressgateway-with-sni-proxy
    12.       enabled: true
    13.       label:
    14.         app: istio-egressgateway-with-sni-proxy
    15.         istio: egressgateway-with-sni-proxy
    16.       k8s:
    17.         service:
    18.           ports:
    19.           - port: 443
    20.             targetPort: 8443
    21.             name: https
    22.         overlays:
    23.         - kind: Deployment
    24.           name: istio-egressgateway-with-sni-proxy
    25.           patches:
    26.           - path: spec.template.spec.containers[-1]
    27.             value: |
    28.               name: sni-proxy
    29.               image: nginx
    30.               volumeMounts:
    31.               - name: sni-proxy-config
    32.                 mountPath: /etc/nginx
    33.                 readOnly: true
    34.               securityContext:
    35.                 runAsNonRoot: true
    36.                 runAsUser: 101
    37.           - path: spec.template.spec.volumes[-1]
    38.             value: |
    39.               name: sni-proxy-config
    40.               configMap:
    41.                 name: egress-sni-proxy-configmap
    42.                 defaultMode: 292 # 0444
    43. EOF

  4. Deploy the new gateway:

    1. $ kubectl apply -f ./egressgateway-with-sni-proxy.yaml

  5. Verify that the new egress gateway is running. Note that the pod has two containers (one is the Envoy proxy and the second one is the SNI proxy).

    1. $ kubectl get pod -l istio=egressgateway-with-sni-proxy -n istio-system
    2. NAME                                                  READY     STATUS    RESTARTS   AGE
    3. istio-egressgateway-with-sni-proxy-79f6744569-pf9t2   2/2       Running   0          17s

  6. Create a service entry with a static address equal to 127.0.0.1 (localhost), and disable mutual TLS for traffic directed to the new service entry:

    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: ServiceEntry
    4. metadata:
    5.   name: sni-proxy
    6. spec:
    7.   hosts:
    8.   - sni-proxy.local
    9.   location: MESH_EXTERNAL
    10.   ports:
    11.   - number: 18443
    12.     name: tcp
    13.     protocol: TCP
    14.   resolution: STATIC
    15.   endpoints:
    16.   - address: 127.0.0.1
    17. ---
    18. apiVersion: networking.istio.io/v1alpha3
    19. kind: DestinationRule
    20. metadata:
    21.   name: disable-mtls-for-sni-proxy
    22. spec:
    23.   host: sni-proxy.local
    24.   trafficPolicy:
    25.     tls:
    26.       mode: DISABLE
    27. EOF

Configure traffic through egress gateway with SNI proxy

  1. Define a ServiceEntry for *.wikipedia.org:

    1. $ cat <<EOF | kubectl create -f -
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: ServiceEntry
    4. metadata:
    5.   name: wikipedia
    6. spec:
    7.   hosts:
    8.   - "*.wikipedia.org"
    9.   ports:
    10.   - number: 443
    11.     name: tls
    12.     protocol: TLS
    13. EOF

  2. Create an egress Gateway for \.wikipedia.org, port 443, protocol TLS, and a virtual service to direct the traffic destined for *.wikipedia.org* through the gateway.

    1. $ kubectl apply -f - <<EOF
    2. apiVersion: networking.istio.io/v1alpha3
    3. kind: Gateway
    4. metadata:
    5.   name: istio-egressgateway-with-sni-proxy
    6. spec:
    7.   selector:
    8.     istio: egressgateway-with-sni-proxy
    9.   servers:
    10.   - port:
    11.       number: 443
    12.       name: tls-egress
    13.       protocol: TLS
    14.     hosts:
    15.     - "*.wikipedia.org"
    16.     tls:
    17.       mode: ISTIO_MUTUAL
    18. ---
    19. apiVersion: networking.istio.io/v1alpha3
    20. kind: DestinationRule
    21. metadata:
    22.   name: egressgateway-for-wikipedia
    23. spec:
    24.   host: istio-egressgateway-with-sni-proxy.istio-system.svc.cluster.local
    25.   subsets:
    26.     - name: wikipedia
    27.       trafficPolicy:
    28.         loadBalancer:
    29.           simple: ROUND_ROBIN
    30.         portLevelSettings:
    31.         - port:
    32.             number: 443
    33.           tls:
    34.             mode: ISTIO_MUTUAL
    35. ---
    36. apiVersion: networking.istio.io/v1alpha3
    37. kind: VirtualService
    38. metadata:
    39.   name: direct-wikipedia-through-egress-gateway
    40. spec:
    41.   hosts:
    42.   - "*.wikipedia.org"
    43.   gateways:
    44.   - mesh
    45.   - istio-egressgateway-with-sni-proxy
    46.   tls:
    47.   - match:
    48.     - gateways:
    49.       - mesh
    50.       port: 443
    51.       sniHosts:
    52.       - "*.wikipedia.org"
    53.     route:
    54.     - destination:
    55.         host: istio-egressgateway-with-sni-proxy.istio-system.svc.cluster.local
    56.         subset: wikipedia
    57.         port:
    58.           number: 443
    59.       weight: 100
    60.   tcp:
    61.   - match:
    62.     - gateways:
    63.       - istio-egressgateway-with-sni-proxy
    64.       port: 443
    65.     route:
    66.     - destination:
    67.         host: sni-proxy.local
    68.         port:
    69.           number: 18443
    70.       weight: 100
    71. ---
    72. # The following filter is used to forward the original SNI (sent by the application) as the SNI of the
    73. # mutual TLS connection.
    74. # The forwarded SNI will be will be used to enforce policies based on the original SNI value.
    75. apiVersion: networking.istio.io/v1alpha3
    76. kind: EnvoyFilter
    77. metadata:
    78.   name: forward-downstream-sni
    79. spec:
    80.   configPatches:
    81.   - applyTo: NETWORK_FILTER
    82.     match:
    83.       context: SIDECAR_OUTBOUND
    84.       listener:
    85.         portNumber: 443
    86.         filterChain:
    87.           filter:
    88.             name: istio.stats
    89.     patch:
    90.       operation: INSERT_BEFORE
    91.       value:
    92.          name: forward_downstream_sni
    93.          config: {}
    94. EOF

  3. Add an EnvoyFilter to the gateway, to prevent it from being deceived.

    1. $ kubectl apply -n istio-system -f - <<EOF
    2. # The following filter verifies that the SNI of the mutual TLS connection is
    3. # identical to the original SNI issued by the client (the SNI used for routing by the SNI proxy).
    4. # The filter prevents the gateway from being deceived by a malicious client: routing to one SNI while
    5. # reporting some other value of SNI. If the original SNI does not match the SNI of the mutual TLS connection,
    6. # the filter will block the connection to the external service.
    7. apiVersion: networking.istio.io/v1alpha3
    8. kind: EnvoyFilter
    9. metadata:
    10.   name: egress-gateway-sni-verifier
    11. spec:
    12.   workloadSelector:
    13.     labels:
    14.       app: istio-egressgateway-with-sni-proxy
    15.   configPatches:
    16.   - applyTo: NETWORK_FILTER
    17.     match:
    18.       context: GATEWAY
    19.       listener:
    20.         portNumber: 443
    21.         filterChain:
    22.           filter:
    23.             name: istio.stats
    24.     patch:
    25.       operation: INSERT_BEFORE
    26.       value:
    27.          name: sni_verifier
    28.          config: {}
    29. EOF

  4. Send HTTPS requests to https://en.wikipedia.org and https://de.wikipedia.org:

    1. $ kubectl exec "$SOURCE_POD" -c sleep -- sh -c 'curl -s https://en.wikipedia.org/wiki/Main_Page | grep -o "<title>.*</title>"; curl -s https://de.wikipedia.org/wiki/Wikipedia:Hauptseite | grep -o "<title>.*</title>"'
    2. <title>Wikipedia, the free encyclopedia</title>
    3. <title>Wikipedia – Die freie Enzyklopädie</title>

  5. Check the log of the egress gateway’s Envoy proxy. If Istio is deployed in the istio-system namespace, the command to print the log is:

    1. $ kubectl logs -l istio=egressgateway-with-sni-proxy -c istio-proxy -n istio-system

    You should see lines similar to the following:

    1. [2019-01-02T16:34:23.312Z] "- - -" 0 - 578 79141 624 - "-" "-" "-" "-" "127.0.0.1:18443" outbound|18443||sni-proxy.local 127.0.0.1:55018 172.30.109.84:443 172.30.109.112:45346 en.wikipedia.org
    2. [2019-01-02T16:34:24.079Z] "- - -" 0 - 586 65770 638 - "-" "-" "-" "-" "127.0.0.1:18443" outbound|18443||sni-proxy.local 127.0.0.1:55034 172.30.109.84:443 172.30.109.112:45362 de.wikipedia.org

  6. Check the logs of the SNI proxy. If Istio is deployed in the istio-system namespace, the command to print the log is:

    1. $ kubectl logs -l istio=egressgateway-with-sni-proxy -n istio-system -c sni-proxy
    2. 127.0.0.1 [01/Aug/2018:15:32:02 +0000] TCP [en.wikipedia.org]200 81513 280 0.600
    3. 127.0.0.1 [01/Aug/2018:15:32:03 +0000] TCP [de.wikipedia.org]200 67745 291 0.659

Cleanup wildcard configuration for arbitrary domains

  1. Delete the configuration items for \.wikipedia.org*:

    1. $ kubectl delete serviceentry wikipedia
    2. $ kubectl delete gateway istio-egressgateway-with-sni-proxy
    3. $ kubectl delete virtualservice direct-wikipedia-through-egress-gateway
    4. $ kubectl delete destinationrule egressgateway-for-wikipedia
    5. $ kubectl delete --ignore-not-found=true envoyfilter forward-downstream-sni
    6. $ kubectl delete --ignore-not-found=true envoyfilter -n istio-system egress-gateway-sni-verifier

  2. Delete the configuration items for the egressgateway-with-sni-proxy deployment:

    1. $ kubectl delete serviceentry sni-proxy
    2. $ kubectl delete destinationrule disable-mtls-for-sni-proxy
    3. $ kubectl delete configmap egress-sni-proxy-configmap -n istio-system
    4. $ kubectl delete -f ./egressgateway-with-sni-proxy.yaml

  3. Remove the configuration files you created:

    1. $ rm ./sni-proxy.conf ./egressgateway-with-sni-proxy.yaml

Cleanup

  • Shutdown the sleep service:

    Zip

    1. $ kubectl delete -f @samples/sleep/sleep.yaml@

  • Uninstall Istio from your cluster:

    1. $ istioctl x uninstall --purge