Thursday, October 8, 2026

Datadog pipeline observability error with source as gcp pub/sub and destination as Rsyslog

 Datadog pipeline observability error with source as gcp pub/sub and destination as Rsyslog

pods panic error before it dies
An error occurred that Vector couldn't handle: receiver disconnected.
The exact panic error
"Tried to call 'Value::as_timestamp_unwrap' on a non-timestamp" exposes a known data-type validation bug inside the Datadog Observability Pipeline Worker's (Vector) engine when format translation is happening
This panic means that the pipeline code is processing a log event and strictly expects its timestamp field to be native Rust/Vector Timestamp metadata object, but it is encountering a different type instead—most likely a raw string or an integer. [1]
Because it calls _unwrap() under the hood without a safe error fallback pattern, the engine panics and causes the GKE pod to crash instantly.

Why did adding DD_OP_DESTINATION_SYSLOG_ENDPOINT_URL trigger this?
When you route a pipeline to a Syslog destination, the Datadog engine automatically injects a hidden Syslog Formatter into the data path. This formatter intercepts incoming events (from your GCP Pub/Sub source) and parses them to build a RFC-5424 or RFC-3164 compliant syslog header, which strictly requires an formatted timestamp. [1]
If your incoming logs from GCP Pub/Sub have overridden, modified, or omitted the root-level @timestampmetadata field (storing it instead as a plain text JSON string), the Syslog translation layer tries to parse it, fails, and panics. [1]

Datadog console Fix
Fix 1: Add a VRL Transform to coerce the Timestamp Type
If you are managing your pipeline components through the Datadog Remote UI (or a local configuration file), you must explicitly force the string field back into a recognized timestamp data type before it gets sent to the rsyslog destination.
Add a Remap / Vector Remap Language (VRL) component right after your GCP Pub/Sub source:
vrl
# Force parse the text timestamp into a proper native timestamp type if exists(.timestamp) {  .timestamp = parse_timestamp!(string!(.timestamp), format: "%Y-%m-%dT%H:%M:%S%.fZ") } else if exists(.@timestamp) {  .@timestamp = parse_timestamp!(string!(.@timestamp), format: "%Y-%m-%dT%H:%M:%S%.fZ") }
Use code with caution.
(Adjust the string format placeholder above to match whatever time string format your GCP logs currently output, e.g., RFC3339).
Fix 2: Drop Custom Mappings on the Datadog Syslog Destination Component
If you configured custom structural attributes under your Syslog Destination Settings via the Datadog UI:
  1. Open the Datadog Observability Pipelines UI dashboard.
  2. Select your pipeline configuration and click edit on your Syslog Destination.
  3. Check the mapping properties. If you customized the Timestamp Key mapping option (e.g., trying to map it to a sub-nested payload key like ://), remove it.
  4. Fall back to letting Datadog use the default root event timestamp metadata.
Fix 3: Change the Syslog Framing Format
Some older rsyslog endpoints struggle with structured RFC-5424 metadata payloads. If the crash persists, change the payload format parameter inside your destination setup from rfc5424 to rfc3164 or use raw text string mode. This changes how the internal encoder evaluates time tokens, bypassing the internal strict unwrap requirements. [1]



Helm Chart fix
To fix this bug when deploying via the Datadog Observability Pipelines Worker (OPW) Helm Chart, you must manually define the pipeline components inside your values.yaml file.
You need to insert a Remap (VRL) transformation block directly between your GCP Pub/Sub source and your Rsyslog destination. This converts the plain text string from GCP into a native timestamp object before the Syslog encoder touches it, completely bypassing the internal Rust panic code.
Here is the exact structure required in your Helm values.yaml configuration:
yaml
# values.yaml pipeline:  # 1. Your existing GCP Pub/Sub Source  sources:    gcp_pubsub_source:      type: "gcp_pubsub"      project: "your-gcp-project-id"      subscription: "your-pubsub-subscription-name"  # 2. THE FIX: Insert this Remap step to cast the type safely  transforms:    fix_timestamp_type:      type: "remap"      inputs:        - "gcp_pubsub_source"      source: |        # Check if timestamp exists and convert it from String to native Timestamp object        if exists(.timestamp) {          parsed, err = parse_timestamp(string!(.timestamp), format: "%Y-%m-%dT%H:%M:%S%.fZ")          if err != null {            # Fallback to the current system time if the GCP format doesn't match perfectly            .timestamp = to_timestamp(now())          } else {            .timestamp = parsed          }        } else {          # Inject a fallback timestamp if the field is entirely missing          .timestamp = to_timestamp(now())        }  # 3. Your Rsyslog Destination (Using the transform as its input)  destinations:    rsyslog_destination:      type: "syslog"      inputs:        - "fix_timestamp_type" # <-- Must pull from the transform, NOT the source directly      endpoint: "rsyslog-service.local:514" # Ensure NO 'tcp://' or 'udp://' prefix      mode: "tcp" # Mode parameter inside the values block works flawless

Friday, May 15, 2026

Datadog product demo

of observability and security platform renowned for providing comprehensive, full-stack visibility across complex IT infrastructures. Its primary strengths lie in consolidating multiple monitoring functions, offering vast integration capabilities, and accelerating incident resolution through AI-powered analytics

Monday, September 29, 2025

Trusted Advisor Adv Management


   Trusted Advisor
   
Deep Dive and Advance Management @Squatch Talks

  • Trusted Advisor Offers a visualization dashboard with QuikSight
  • Trusted Advisor APIs can be used to get a list of checks, recommendations, and resources for recommendations.
  • You can use TA APIs for your infrastructure automation and audits you want to build



My running experience


    My running story, a personal goal 

Sunday, September 28, 2025

ebsnvme-id output not consistent

[root]# rpm -qf /sbin/ebsnvme-id ec2-utils-1.0-2.amzn2.noarch
[root]# ebsnvme-id -b /dev/nvme0n1 xvda

[root]# ebsnvme-id -b /dev/nvme1n1 /dev/xvde
[root]# ebsnvme-id -b /dev/nvme2n1 /dev/xvdf

 We have discovered that if the disks are attached after the instance is created, the command will return the full path. If the disks are attached and then the instance is rebooted for example, the command will prune off the /dev/ prefix. This seems to be a problem with whether or not the ebs volumes are present in the instance metadata. I rebooted one of the instances we were working with yesterday and now the output is reflecting this new output:

[root@vlt-us-west-2-production-1106-solr-1 ~]# uptime
13:02:44 up 2 min, 1 user, load average: 0.33, 0.28, 0.11

[root@vlt-us-west-2-production-1106-solr-1 ~]# /sbin/ebsnvme-id -b /dev/nvme2n1 xvde
instance doesn't know abt the ebs volume, it was added later

[root@vlt-eu-central-1-production-2047-solr-6 ~]# curl -m3 http://169.254.169.254/latest/meta-data/block-device-mapping/ami

[root@vlt-us-west-2-production-1106-solr-1 ~]# curl -m3 http://169.254.169.254/latest/meta-data/block-device-mapping/ http://169.254.169.254/latest/meta-data/block-device-mapping ami 

 ebs14 ebs2

AWS Organization Policies and AWS Config Rules


How to enforce and evaluate resource tagging


AWS Organization Tag Policy: How to Limit/Control what the end user can select when they are required to provide a Tag Key and Tag Value for any resource deployment.

AWS Organization APIs Tag Policy: How to create the Tag Policy using AWS Organization APIs.  Integrate department values from external sources like SAP and ServiceNow to create and automate the process.

AWS Config: Use AWS Config to assess, audit, and evaluate the values of your AWS resources. In the case of tagging, provide resource compliance monitoring/remediation to identify resources that are lacking tags with specific keys, using the required_tags rule.

Wednesday, July 27, 2022

Gitlab 500 internal server error while accessing repos

came across this issue when restoring gitlab 8.16.4-ee to a DR site

ERROR : 500 internal server error

while accessing repo
tail -f /var/log/gitlab/gitlab-rails/production.log

OpenSSL::Cipher::CipherError (bad decrypt):
  app/models/project.rb:519:in `import_url'
  app/models/project.rb:555:in `external_import?'
  app/models/project.rb:547:in `import?'
  app/models/project.rb:563:in `import_in_progress?'
  app/controllers/projects_controller.rb:95:in `show'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'
  lib/gitlab/middleware/readonly_geo.rb:29:in `call'


SOLUTION
It is pretty obvious that it is related to the gitlab secret key

"OpenSSL bad decrypt"

#1. copy /etc/gitlab/gitlab-secrets.json from old instance to new
#2. sudo gitlab-rails runner  "Project.where.not(import_url: nil).each { |p| p.import_data.destroy if p.import_data }"


Gitlab Backups:
## gitlab-rake gitlab:backup:create

You may run into this error -
Dumping PostgreSQL database gitlab_prod ... pg_dump: server version: 9.5.2; pg_dump version: 9.2.18)

just make sure you point pg_dump and psql to latest version

/opt/gitlab/embedded/bin/pg_dump --version
pg_dump (PostgreSQL) 9.6.1

lrwxrwxrwx 1 root root   49 Jan 31 17:10 pg_dump -> /opt/gitlab/embedded/postgresql/9.6.1/bin/pg_dump

lrwxrwxrwx 1 root root       46 Jan 31 17:42 psql -> /opt/gitlab/embedded/postgresql/9.6.1/bin/psql


## make sure you backup /etc/gitlab/gitlab-secrets.json and /etc/gitlab/gitlab.rb seperately

Wednesday, December 1, 2021

AWS SysOps certification story

 My AWS SysOps certification story:

10/10/21 AWS Certified Solutions Architect - Associate (SAA-C02) - cleared thru PSI
Last time I did this certification was way back in march 2017

10/12/21 AWS Certified Cloud Practitioner (CLF-C01), cleared thru PSI
Actually wanted to appear for SysOps-Associate.
Requested a voucher for PSI and then figure out the PSI doesn't offer SysOps-Associate test.
I believe that's the only test PSI doesn't offer.

10/18/21 AWS Certified SysOps Administrator - Associate (SOA-C02) - failed thru Pearson Vue
This was very interesting. I got 55 questions, I was done with all the questions and I had more 70 mins left. I was thinking may be this is the easiest exam comparatively. I took my own time to revise all the questions twice. I had about 40mins and decided to submit/close it. Once I submitted it said you have 3 labs for 20mins each and you cannot go back to the lab once submitted. I was not prepared for these labs and had not idea that this test had labs.
I somehow started with the lab, couldn't able to follow all the steps in the first lab. I had only 20mins left for the next two labs
Luckily the 2nd and 3rd labs were easy enough and still had 9 mins left but couldn't go back to the first to fix it.

So, there goes my first failed attempt and now I have wait for 2 weeks before I can attempt this test again. Apparently this is the only test that has labs as of now

Exam labs make up 20% of your total score. A passing score is 720 out of 1,000.

Another thing about this test, or rather not good and frustrating about this test is - it doesn't tell you if you passed or failed right away. It takes couple of days, as per the official statement, it may take upto 5 business days. Most likely because of labs, someone has to review the labs steps and sanctity.

11/18/21 AWS Certified SysOps Administrator - Associate (SOA-C02) - no show thru Pearson Vue
I had to reschedule my test multiple times due to conflict with other meetings and things to do.
It was a 8am test, I login to start the test and find out there is nothing for me to start. I panicked, started calling the Pearson support number (866-207-9983)
They told me, this test was scheduled for 11/12/21 and I didn't show up :-)
I however reschedule my 11/12/21 test to 11/18/21 but it didn't go thru or not sure if I never clicked submit to finalize.
Luckily I had captured the snapshot of the last page of exam time. 
I created a ticket with Pearson Vue Case: 07312581, and after few email exchange they cancelled that exam and sent me the refund and restored my voucher.
Please always make sure you have the confirmation email of the exam and also login once before the exam and check the schedule is updated.


11/19/21 AWS Certified SysOps Administrator - Associate (SOA-C02) - failed/void thru Pearson Vue
This test went well, unless I reached lab2.
In Lab2 I couldn't able to edit values in the template, was not able to select the words properly to edit.
Couldn't able type numbers in the notepad and many other issues.
Created a support case with AWS CASE 9232500091, they asked me to create a case with Pearson Vue.
Created a case with Pearson Case ID: 07321951

The response from Pearson after few days of review.
"First and foremost, we want to sincerely apologize for any inconvenience you may have faced during your exam.
A case has been submitted on your behalf to AWS, and you can expect to receive an update from AWS shortly.
Our team would like to fix this by offering you a customer service voucher code that can be used to schedule
your next exam at no additional charge to you. "

Once they make the exam void and 2 weeks wait period doesn't apply.


11/30/21
- AWS Certified SysOps Administrator - Associate (SOA-C02) - cleared thru Pearson Vue
Finally I was able re-schedule my 12/3/21 to 11/30/21 to finish it

Bottomline, it creates a huge bump in planning ahead if you fail the test or do not pay enough attention to scheduling or test pattern details. And other thing is this can be a very difficult exam if you are NOT hands-on. The labs are not very difficult but it has its own performance and layout challenges when you are working thru the restricted browser.  And finally I would recommend tutorialsdojo.com for labs

Hope you will not make these mistakes I made

Tuesday, April 27, 2021

ansible error with python boto3

 Ansible Error:

TASK [provision-ai-vpc : Create VPC] ************************************* task path: /Users/pankajgautam/infra/ansible/roles/provision-ai-vpc/tasks/main.yml:14 redirecting (type: modules) ansible.builtin.ec2_vpc_net to amazon.aws.ec2_vpc_net fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (botocore or boto3) on Veevas-MacBook-Pro.local's Python /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"} PLAY RECAP ********************************************************************* localhost : ok=1 changed=0 unreachable=0 failed=1 skipped=3 rescued=0 ignored=0 


It seems like Mac comes with python2.7 and cannot be touched/removed

pankajgautam@Veevas-MBP ~ % /usr/bin/python --version

Python 2.7.16


If you install awscli it comes with its own python

pankajgautam@Veevas-MBP ~ % /usr/local/aws-cli/aws --version

aws-cli/2.1.37 Python/3.8.8 Darwin/20.4.0 exe/x86_64 prompt/off


You can install python with pip or pip3 that generally installs under /Library

pankajgautam@Veevas-MBP ~ % ls  /Library/Python/ 

2.7 3.8


Then you can also install python using brew that gets installed under /usr/local/Cellar

pankajgautam@Veevas-MBP ~ % ls -l /usr/local/Cellar/ | grep python

drwxr-xr-x  3 pankajgautam  admin  96 Apr 21 12:58 python@3.9




You can specify which python to be used with ansible using ansible_python_interpreter

and also make sure you have boto3 module imported in that python version

 

AWS_PROFILE=aiprod ansible-playbook -e "aws_region=us-east-1 release_type=gr ansible_python_interpreter=/usr/local/Cellar/python@3.9/3.9.4/bin/python3" -i ansible/inventory/prod ansible/provision-ai-vpc-pb.yml -vv 2>&1 | tee provision.log 


pankajgautam@Veevas-MBP % sudo pip3 install boto3 

WARNING: The directory '/Users/pankajgautam/Library/Caches/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. Collecting boto3 Downloading boto3-1.17.54-py2.py3-none-any.whl (131 kB) |████████████████████████████████| 131 kB 640 kB/s 

Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.9/site-packages (from boto3) (0.10.0) 

Collecting botocore<1.21.0,>=1.20.54 

Downloading botocore-1.20.55-py2.py3-none-any.whl (7.4 MB) |████████████████████████████████| 7.4 MB 545 kB/s Collecting s3transfer<0.5.0,>=0.4.0 Downloading s3transfer-0.4.1-py2.py3-none-any.whl (79 kB) |████████████████████████████████| 79 kB 16.1 MB/s Collecting urllib3<1.27,>=1.25.4 Downloading urllib3-1.26.4-py2.py3-none-any.whl (153 kB) |████████████████████████████████| 153 kB 18.2 MB/s Collecting python-dateutil<3.0.0,>=2.1 Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB) |████████████████████████████████| 227 kB 16.1 MB/s Collecting six>=1.5 

Downloading six-1.15.0-py2.py3-none-any.whl (10 kB) Installing collected packages: six, urllib3, python-dateutil, botocore, s3transfer, boto3 Successfully installed boto3-1.17.54 botocore-1.20.55 python-dateutil-2.8.1 s3transfer-0.4.1 six-1.15.0 urllib3-1.26.4 


pankajgautam@Veevas-MBP infra % /usr/local/Cellar/python@3.9/3.9.4/bin/python3

Python 3.9.4 (default, Apr 5 2021, 01:49:30) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin Type "help", "copyright", "credits" or "license" for more information. 

>>> import boto3 >>> 


Thursday, February 6, 2020

migrating to (aws nitro based system) machine type m4 to m5


Tested the conversion with RHEL8 and AmzonLinx2
from m4=>m5 and r4=>r5


As of now following instances are based on the Nitro system:
A1, C5, C5d, C5n, G4, I3en, Inf1, M5, M5a, M5ad, M5d, M5dn, M5n, p3dn.24xlarge, R5, R5a, R5ad, R5d, R5dn, R5n, T3, T3a, and z1d

Before changing your instance to a nitro based system, make sure
1. The (ENA) elastic network adapter is installed and enabled for the instance.
2. The NVMe driver is installed on the instance and is loaded in the initramfs image of the instance.
3. Use "/etc/fstab" to mount the file systems using UUID/Label.

EBS volumes
[root@pod ~]# lsblk
NAME    MAJ:MIN  RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0     0  200G  0 disk
├─xvda1 202:1     0    1M  0 part
└─xvda2 202:2     0  200G  0 part /
xvdd    202:48    0  500G  0 disk /data
xvde    202:80    0  500G  0 disk /data/logs
xvdf    202:160   0  500G  0 disk /data/active

NVMe EBS volume
[root@pod ~]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1     259:8    0  200G  0 disk
├─nvme0n1p1 259:9    0    1M  0 part
└─nvme0n1p2 259:10   0  200G  0 part /
nvme1n1     259:3    0  500G  0 /data
nvme2n1     259:4    0  500G  0 /data/logs
nvme3n1     259:2    0  500G  0 /data/active

NVMe map command:
# /sbin/ebsnvme-id /dev/nvme1n1
Volume ID: vol-0fb8db1fddd1f5834
xvde

Suggested /etc/fstab
UUID=35e495d3-a1df-40d9-bb6c-e37b81aec11f /data xfs noatime 0 0
UUID=075bd9b0-370e-48b1-9de2-4d061d16ca5b /data/logs xfs noatime 0 0
UUID=5e5fde64-1346-45e0-9eeb-6c68be8b81bd /data/active xfs noatime 0 0

Commands to check instance if its ready to changed to nitro system

lsinitrd /boot/initramfs-$(uname -r).img|grep nvme
modinfo nvme
modinfo ena
dracut -f -v



Lesson learned:

Currently AWS takes care of these volume device name change from 4 series instance to 5 series nitro based instances on the amazon linux 2 only. And the same steps should also work with Redhat hosts if we apply the udev rule and python script.

With amazon linux 2 the magic code that is creating the symlinks from /dev/xvdb, or /dev/xvdc devices to /dev/nvme1n1 or /dev/nvme2n1 is by the udev rules

In this file "/etc/udev/rules.d/70-ec2-nvme-devices.rules",

and more specifically this line KERNEL=="nvme[0-9]*n[0-9]*", ENV{DEVTYPE}=="disk", ATTRS{model}=="Amazon Elastic Block Store", PROGRAM="/sbin/ebsnvme-id -u /dev/%k", SYMLINK+="%c"
If we read that - this is saying, when a device is attached to the system, that has a kernel device name that matches the pattern given, run this program and create a symlink with the output of that program.

Is a python script (/sbin/ebsnvme-id) to get the underlying mount information then to create a sym link, 


One thing to keep in mind here is that this only works if the device presented to amazon linux 2 hosts are in /dev/xvdX format and NOT /dev/sdb or /dev/sdc which is an old format. In that case you may have tweak the rules to get the results you desire


Recommendation from  c5_m5_checks_script
OK     NVMe Module is installed and available on your instance
OK     ENA Module with version 2.0.2K is installed and available on your instance


Printing correct fstab file below:
# /etc/fstab
UUID=a727b695-0c21-404a-b42b-3075c8deb6ab /                       xfs     defaults        0 0
UUID=587cde86-c167-4e73-92cb-b67739d9991d /data/logs xfs  noatime 0 0
UUID=6410c3de-3714-4198-b31b-afeca374ef43 /data/active xfs  noatime 0 0

Tuesday, November 19, 2019

RequestsDependencyWarning urllib3 (1.25.6) or chardet (3.0.4) doesn't match a supported version

edit  /usr/lib/python3/dist-packages/requests/__init__.py


 # Check urllib3 for compatibility.
    major, minor, patch = urllib3_version  # noqa: F811
    major, minor, patch = int(major), int(minor), int(patch)
    # urllib3 >= 1.21.1, <= 1.22
    assert major == 1
    assert minor >= 21
###    assert minor <= 22
    assert minor <= 25                        <<<<<<<<<<===================== new line



    # Check chardet for compatibility.
    major, minor, patch = chardet_version.split('.')[:3]
    major, minor, patch = int(major), int(minor), int(patch)
    # chardet >= 3.0.2, < 3.1.0
    assert major == 3
    assert minor < 1
    assert patch >= 2

Friday, November 1, 2019

CDN requirements


Few CDN requirements that we would like to see and test right away.

#1. cross domain functionality
#2. auth token for all assets
#3. exclusion rules for file extension and urls
#4. compression on large content
#5. compatibility with jwplayer to invoke media content
#6. able to transcode media streaming into m3u8 format
#7. various traffic reporting on each cnames
#8. caching rules for nodes and expiration policies
#9. APIs for accessing origin server
#10. Able to map cnames to a specific path

AWS EKS ~/.kube/config (heptio vs awscli)

You can config your kubectl to talk to EKS both ways

AWSCli Token
pankaj@PGautam-X280:~/terraform/eks3-thar$ aws eks get-token --cluster-name thar-eks-dev-cluster-2
{"kind": "ExecCredential", "apiVersion": "client.authentication.k8s.io/v1alpha1", "spec": {}, "status": {"expirationTimestamp": "2019-11-01T21:59:14Z", "token": "k8s-aws-v1.aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8_QWN0aW9uPUdldENhbGxlcklkZW50aXR
5JlZlcnNpb249MjAxMS0wNi0xNSZYLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZY
Q1MTRaJlgtQW16LUV4cGlyZXM9NjAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JTNCeC1rO
HMtYXdzLWlkJlgtQW16LVNpZ25hdHVyZT0yNTdmNTE4YjBjOWNjZGE4MmNmOTk1YWI2ND
FmYzZjZTM2YTdhYTZiM2UwNjBmZGUwZDg3ZTMzMWQ1NGYyODQ5"}}


Heptio Token
pankaj@PGautam-X280:~/terraform/eks3-thar$ aws-iam-authenticator token -i eks-thar-techops-dev-cluster-1
{"kind":"ExecCredential","apiVersion":"client.authentication.k8s.io/v1alpha1","spec":{},"status":{"token":"k8s-aws-v1.aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8_QWN0aW9uPUdldENhbGxlcklkZW50aX
R5JlZlcnNpb249MjAxMS0wNi0xNSZYLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYL
JlgtQW16LUV4cGlyZXM9NjAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JTNCeC1rOHMtYXdzL
WlkJlgtQW16LVNpZ25hdHVyZT00M2Q2NmUzNWI4ZWNlNDExNzEwYmMxOTNjNzI4YmEzMGM
wN2E2ZjZjNjRmZDIxN2Y1ZjYwODk0NGNmZmMwM2Zk"}}


AWSCli configuration file:

apiVersion: v1
clusters:
- cluster:
    server: https://093E146E63E428CC642F01362F6BE661.sk1.us-west-2.eks.amazonaws.com
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURB
ZtQm1WQ1U4OXZDamZpOXBNaQo2Nmp3NkxTWSttbnFQYzhVcUdVbks5WURhZU01b0ZvULQo=
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws
      env:
      - name: AWS_PROFILE
        value: opssbx
      args:
      - "eks"
      - "get-token"
      - "--cluster-name"
      - "thar-eks-dev-cluster-2"
      # - "--role"
      # - "arn:aws:iam::191829356374:role/ops"

 
Heptio configuration file:
apiVersion: v1
clusters:
- cluster:
    server: https://093E146E63E428CC642F01362F6BE661.sk1.us-west-2.eks.amazonaws.com
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURB
TkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHV
npNQjRYRFRFNU1URXdNVEU1TURJMU1Wb1hEVEk1TVRBeU9URTVNREkxTVZvd0ZURVR
NQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQl
GSUNBVEUtLS0tLQo=
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      env:
      - name: AWS_PROFILE
        value: opssbx
      args:
        - "token"
        - "-i"
        - "thar-eks-dev-cluster-2"

Thursday, October 31, 2019

AWS EKS cluster authentication Unauthorized

Scenario: After you created a EKS cluster with worker nodes using Terraform. You are not able to connect to the cluster endpoint. The cluster was created from an account which has admin rights to assume roles in other accounts. We confirmed the authorized roles and policies are correct. This terraform was used multiple times to create a dev cluster, however this time we are creating the cluster in a different account.
It seems like the EKS is not able to validate the user, or we are not able to send the right user/credential to the cluster. So its not an authorization issue but an authentication problem.
So, we have explore more into authentication modules rather then EKS role based access control (RBAC)

One of the first thing you can do is enable cluster auth logs and see which user cloudwatch is capturing as access denied.

Errors:

pankaj@PGautam-X280:~/terraform/eks3-thar$ kubectl versionClient Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
error: You must be logged in to the server (the server has asked for the client to provide credentials)


pankaj@PGautam-X280:~/terraform/eks3-thar$ kubectl get nodes
error: the server doesn't have a resource type "nodes"


pankaj@PGautam-X280:~/terraform/eks3-thar$ kubectl version --v=7

I1030 15:55:05.651723   23213 loader.go:359] Config loaded from file /home/pankaj/.kube/config
I1030 15:55:05.653152   23213 round_trippers.go:383] GET https://A4A4CXXX0XXXXXXXE.gr7.us-west-2.eks.amazonaws.com/version?timeout=32s
I1030 15:55:05.653363   23213 round_trippers.go:390] Request Headers:
I1030 15:55:05.653585   23213 round_trippers.go:393]     Accept: application/json, */*
I1030 15:55:05.653855   23213 round_trippers.go:393]     User-Agent: kubectl/v1.11.2 (linux/amd64) kubernetes/bb9ffb1
I1030 15:55:06.023642   23213 round_trippers.go:408] Response Status: 401 Unauthorized in 369 milliseconds
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
I1030 15:55:06.035173   23213 helpers.go:201] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server has asked for the client to provide credentials",
  "reason": "Unauthorized",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "unknown"
      }
    ]
  },
  "code": 401

}]
F1030 15:55:06.036149   23213 helpers.go:119]
error: You must be logged in to the server (the server has asked for the client to provide credentials)


=========================================
Solution:

Cluster is active
pankaj@PGautam-X280:~/terraform/eks3-thar$ aws eks --profile ops --region us-west-2 describe-cluster --name eks-dev-cluster-1 --query cluster.status
"ACTIVE"

Heptio token works:
pankaj@PGautam-X280:~/terraform/eks3-thar$ aws-iam-authenticator token -i eks-dev-cluster-1
{"kind":"ExecCredential","apiVersion":"client.authentication.k8s.io/v1alpha1","spec":{},"status":{"token":"k8s-aws-v1.aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8_QWN0aW9uPUdldENhbGxlcklkZ
W50aXR5JlZlcnNpb249MjAxMS0wNi0xNSZYLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNI
QTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFaUkhORE9GVzJPM0ZXUjdIJTJGMjAxOTEw
N0JTNCeC1rOHMtYXdzLWlkJlgtQW16LVNpZ25hdHVyZT1jZjk4NTg5MDI2ZjA2MjMyM2Y5Njh
iNDljMzI2YWZmYmI5MDcxNzBlMTUxZDNlN2QxYzA0NGFiOWE0MDcxNTc3"}}

Update .kube config with env:
pankaj@PGautam-X280:~/terraform/eks3-thar$ echo $KUBECONFIG
/home/pankaj/terraform/eks3-thar/config

- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      env:
      - name: AWS_PROFILE
        value: opssbx

      args:
        - "token"
        - "-i"
        - "eks-dev-cluster-1"


pankaj@PGautam-X280:~/terraform/eks3-thar$ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6-eks-5047ed", GitCommit:"5047edce664593832e9b889e447ac75ab104f527", GitTreeState:"clean", BuildDate:"2019-08-21T22:32:40Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

Tuesday, January 15, 2019

AWS EKS - elb in pending state

## ELB Error ( in pending state)
Error creating load balancer (will retry): failed to ensure load balancer for service default/hellopankajgautam:

Solution: Tagging required for VPC/Subnets and private
https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html

"kubernetes.io/cluster/${var.product}-${var.env}-${var.cluster_name}" = "shared"
"kubernetes.io/role/internal-elb" = "1"

in my experience so far - the only time elb is in pending state -
    if there not tagging available as mentioned above
    or there is no ips left
    or if there is any soft limit errors

Thursday, March 23, 2017

cloudformation, cloud-init, cfn-init with redhat 7

I started playing with CloudFormation, Designer a month back and so far it is working out pretty well to all my requirements.

## CloudFormation Designer is pretty nice which allows you can visualize the whole architecture in one frame. It is also very useful and handy if you would like to review just one resource, just select the resource and see code in component tab.
However its kind of painful when the drawing goes haywire and you have to stretch it back to same proportion. Sometime you can cannot select a component directly to drag/move it around. Another problem is with JSON validation. It gives some strange co-ordinates when there is an improper syntax, instead pointing out the line number.

If you are in the same show and looking for a tool to validate JSON,
Please look at http://jsonlint.com/


## It's comparatively very easy to start with CF and deploy a simple infrastructure. For example you can create a VPC, public subnet, private subnet, security groups, Internet gateway, Route table, Network ACLs etc. You can do advance setup like auto scaling, ELB etc also. It may take some time you work with JSON again to figure out EC2 properties like Network interfaces, Network ACL rules,  etc. How depends on works?

Some of the cryptic errors are "Device property cannot be empty". Make sure you review the whole template and see what doesn't make sense. Also the "Events" tab while template is running gives you enough clue what's going on. I don't know if there is any way to try out some of these commands without running the entire template.


## Once you have mastered infrastructure code, then comes application provisioning and deployment. And that's where most of the cloud-init and cfn-init mostly comes into play.
When you can start a EC2 you can do few automation with "UserData", but if you have to do a whole bunch of the things at the time of creation, "UserData" becomes nightmare because the way it reads the syntax.

## cloud-init
Cloud-init will execute whatever is defined in "UserData". For example if you want to get object from private S3 and execute a script or copy a config.
You can execute everything exposing your  AWS Access key and Secret key like we do using CLI in the CF template.
Best practice is to use role for EC2 using Cloudformation Authentication where you only allow the instance to authenticate to the private S3 and download config or execute script using cfn-init.

## cfn-init
For some reason its not very straight forward and documented well if you have to install cfn-init on redhat, as of now atleast.

=====================================================================
"UserData": {
          "Fn::Base64": {
          "Fn::Join": [ "", [ "#!/bin/bash -ex", "\n",
          "cd /root \n",
          "yum -y install unzip wget dos2unix git bind-utils ec2-net-utils \n",
          "curl https://s3.amazonaws.com/aws-cli/awscli-bundle.zip -o awscli-bundle.zip \n",
          "unzip awscli-bundle.zip \n",
          "awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws \n",
           "python awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws \n",
           "curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py \n",
           "python get-pip.py \n",
           "pip install pystache \n",
           "pip install argparse \n",
           "pip install python-daemon \n",
           "pip install requests \n",
           "cd /opt \n",
           "curl https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz -o aws-cfn-bootstrap-latest.tar.gz \n",
            "tar -xvpf aws-cfn-bootstrap-latest.tar.gz \n",
            "cd aws-cfn-bootstrap*/ \n",
            "python setup.py build \n",
            "python setup.py install \n",
            "cd /opt \n",
            "mkdir aws \n",
            "cd aws \n",
            "mkdir bin \n",
            "ln -s /usr/bin/cfn-hup /opt/aws/bin/cfn-hup \n",
            "ln -s /usr/bin/cfn-init /opt/aws/bin/cfn-init \n",
            "/opt/aws/bin/cfn-init -v ",
            "         --stack ", { "Ref" : "AWS::StackName" },
            "         --resource upsrcinst1",
            "         --region ", { "Ref" : "AWS::Region" }, "\n",
            "dos2unix /script.sh \n",
            "sh /

script
.sh \n", "\n"
                     ] ]
                 }
             },
                "Tags": [ { "Key": "Name", "Value": "app-server" } ]

              },


Reference:
https://aws.amazon.com/blogs/devops/authenticated-file-downloads-with-cloudformation/

These are the 3 items you have to include in CF script.

1. Create IAM Roles/Policies
Create InstanceRole
Create RolePolicies
Create InstanceProfile



(It will auto create a Role XXXX-InstanceRole-1H2PIS782ZM81)

Under Instance resource:
2. Include in instance metadata:

Create CloudFormation Authentication
Create Cloudformation Init

3. Include IAM Instance profile in instance properties:
"IamInstanceProfile":{ "Ref":"InstanceProfile" },

=====================================================================
"InstanceRole":{
       "Type":"AWS::IAM::Role",
       "Properties":{
             "AssumeRolePolicyDocument":{
              "Statement":[ { "Effect":"Allow", "Principal":{ "Service":[ "ec2.amazonaws.com" ] },  "Action":[  "sts:AssumeRole"  ] } ]
             },
             "Path":"/"
        }
    },


    "RolePolicies":{
         "Type":"AWS::IAM::Policy",
         "Properties":{
              "PolicyName":"S3Download",
              "PolicyDocument":{
                 "Statement":[ {  "Action":[ "s3:GetObject" ], "Effect":"Allow",           "Resource":"arn:aws:s3:::bucket/bucketfolder/*" } ] 
             },
             "Roles":[  { "Ref":"InstanceRole"   } ]
          }
    },


    "InstanceProfile":{
        "Type":"AWS::IAM::InstanceProfile",
        "Properties":{
             "Path":"/",
             "Roles":[  {  "Ref":"InstanceRole"  }  ]
         }
    },


 ====================================================================

"Metadata": {
           "AWS::CloudFormation::Designer": { "id": "85a7776e-769f-4f98-85e0-87e7b58f381f" },

           "AWS::CloudFormation::Authentication":{ "S3AccessCreds":{ "type":"S3", "roleName":{ "Ref":"InstanceRole"  }, "buckets" : ["bucket"]     }  },


              "AWS::CloudFormation::Init": { "config": { "files": { "/root/script.sh": { "source": "https://s3-us-west-2.amazonaws.com/bucket/bucketfolder/script.sh", "mode":"000755", "owner":"ec2-user", "authentication":"S3AccessCreds"  } } } }

        }


=====================================================================
 

"inst1": {
        "Type": "AWS::EC2::Instance",
        "DependsOn": "upsrcebs1",
        "Properties": {
               "IamInstanceProfile":{ "Ref":"InstanceProfile" },

----
----
}

=====================================================================

Thursday, October 27, 2016

mysql replication and verification steps

Imp things to remember:

#### slave entries:
relay-log=mysqld-relay-bin
report-host=hostname

### ssh tunnel with keys:

back01# mysql

    mysql> show databases;

    +--------------------+
    | Database           |
    +--------------------+
    | information_schema |
    | mysql              |
    | test               |
    | mywork_ops_backups  |
    +--------------------+

    4 rows in set (0.00 sec)



backup/dump

mysqldump mywork_ops_backups < /database_backups/myworkopsdb-$(date +%m-%d-%Y).sql


restore

mysql> create database ops_backup;
[root@pgvmdc ~]# mysql ops_backup < /root/pg-myworkopsdb-20141107.sql

mysql> describe backup_job;
+-----------------+-----------------------------------------------------+------+-----+---------+----------------+
| Field           | Type                                                | Null | Key | Default | Extra          |
+-----------------+-----------------------------------------------------+------+-----+---------+----------------+
| backup_id       | int(10) unsigned                                    | NO   | PRI | NULL    | auto_increment |
| job_start_dt_tm | datetime                                            | NO   |     | NULL    |                |
| job_end_dt_tm   | datetime                                            | NO   |     | NULL    |                |
| pod_name        | varchar(64)                                         | NO   |     | NULL    |                |
| status          | enum('Success','Error','Running','Rerun','md5fail') | NO   |     | NULL    |                |
| type            | enum('inc','full')                                  | NO   |     | NULL    |                |
| dateb           | varchar(64)                                         | NO   |     | NULL    |                |
| filename        | varchar(64)                                         | NO   |     | NULL    |                |
| filesize        | varchar(64)                                         | YES  |     | NULL    |                |
| vault           | varchar(64)                                         | NO   |     | NULL    |                |
+-----------------+-----------------------------------------------------+------+-----+---------+----------------+
10 rows in set (0.00 sec)




select * from backup_job where status ='Error';



mysql> select count(*) from backup_job;
+----------+
| count(*) |
+----------+
|    75977 |
+----------+

1 row in set (0.01 sec)


edit master /etc/my.cnf and restart


##replication master##

server-id=1

log-bin=mysql-bin

report-host=hostname

##replication master##

auto_increment_increment=2

auto_increment_offset=1


edit slave /etc/my.cnf and restart


##replication slave##

server-id=2

relay-log=mysqld-relay-bin

report-host=hostname

##replication slave##

auto_increment_increment=2

auto_increment_offset=2



replication user on master

mysql> create user vrep identified by 'vrepvrep';
Query OK, 0 rows affected (0.00 sec)


mysql> create user vrep identified by '2014#vrep';
Query OK, 0 rows affected (0.03 sec)

mysql> select host, user, password from mysql.user;

+---------------------+------+-------------------------------------------+
| host                | user | password                                  |
+---------------------+------+-------------------------------------------+
| localhost           | root |                                           |
| pgvmdc.myworkdev.com | root |                                           |
| 127.0.0.1           | root |                                           |
| localhost           |      |                                           |
| pgvmdc.myworkdev.com |      |                                           |
| %                   | vrep | *63536F52A6C2FC15C6CBECB7ADB3F9299D86D6B3 |
+---------------------+------+-------------------------------------------+

6 rows in set (0.00 sec)


mysql> grant replication slave on *.* to 'vrep'@'%' identified by 'vrepvrep';
Query OK, 0 rows affected (0.00 sec)


or


mysql> grant replication slave on *.* to vrep identified by 'vrepvrep';
Query OK, 0 rows affected (0.00 sec)


mysql> set password for 'vrep' = password('vrep');
Query OK, 0 rows affected (0.00 sec)


find master co-ordinates

mysql> flush tables with read lock;
mysql> show master status;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000001 |      320 |              |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)



Setting the Master Configuration on the Slave:
mysql> change master to master_host='mysqlmaster';
Query OK, 0 rows affected (0.02 sec)

mysql> change master to master_user='vrep';
Query OK, 0 rows affected (0.01 sec)

mysql> change master to master_password='vrep';
Query OK, 0 rows affected (0.01 sec)

mysql> change master to master_log_file='vrep.log';
Query OK, 0 rows affected (0.00 sec)

mysql> change master to master_log_pos=320;
Query OK, 0 rows affected (0.00 sec)

mysql> start slave;
Query OK, 0 rows affected (0.00 sec



##################################registration error###################################

mysql> show slave status\G

*************************** 1. row ***************************

               Slave_IO_State:
              Master_Host: mysqlmaster

                  Master_User: vrep

                  Master_Port: 3306

                Connect_Retry: 60

             Master_Log_File: vrep.log

          Read_Master_Log_Pos: 1147

               Relay_Log_File: mysqld-relay-bin.000001

                Relay_Log_Pos: 4

        Relay_Master_Log_File: vrep.log

             Slave_IO_Running: No

            Slave_SQL_Running: No

              Replicate_Do_DB:

          Replicate_Ignore_DB:

           Replicate_Do_Table:

       Replicate_Ignore_Table:

      Replicate_Wild_Do_Table:

  Replicate_Wild_Ignore_Table:

                   Last_Errno: 0

                   Last_Error:

                 Skip_Counter: 0

          Exec_Master_Log_Pos: 1147
              Relay_Log_Space: 106

              Until_Condition: None

               Until_Log_File:

                Until_Log_Pos: 0

           Master_SSL_Allowed: No

           Master_SSL_CA_File:

           Master_SSL_CA_Path:

              Master_SSL_Cert:

            Master_SSL_Cipher:

               Master_SSL_Key:

        Seconds_Behind_Master: NULL

Master_SSL_Verify_Server_Cert: No

                Last_IO_Errno: 1597

                Last_IO_Error: Master command COM_REGISTER_SLAVE failed: Access denied for user 'vrep'@'mysqlslave' (using password: YES) (Errno: 1045)

               Last_SQL_Errno: 0

               Last_SQL_Error:

1 row in set (0.00 sec)





##################################connection error#####################################

mysql> show slave status\G

*************************** 1. row ***************************

               Slave_IO_State:

                  Master_Host: mysqlmaster

                  Master_User: vrep

                  Master_Port: 3306

                Connect_Retry: 60

              Master_Log_File: mysql-bin.000004

          Read_Master_Log_Pos: 366

               Relay_Log_File: mysqld-relay-bin.000001

                Relay_Log_Pos: 4

        Relay_Master_Log_File: mysql-bin.000004

             Slave_IO_Running: No

            Slave_SQL_Running: No

              Replicate_Do_DB:

          Replicate_Ignore_DB:

           Replicate_Do_Table:

       Replicate_Ignore_Table:

      Replicate_Wild_Do_Table:

  Replicate_Wild_Ignore_Table:

                   Last_Errno: 0

                   Last_Error:

                 Skip_Counter: 0

          Exec_Master_Log_Pos: 366

              Relay_Log_Space: 106

              Until_Condition: None

               Until_Log_File:

                Until_Log_Pos: 0

           Master_SSL_Allowed: No

           Master_SSL_CA_File:

           Master_SSL_CA_Path:

              Master_SSL_Cert:

            Master_SSL_Cipher:

               Master_SSL_Key:

        Seconds_Behind_Master: NULL

Master_SSL_Verify_Server_Cert: No

                Last_IO_Errno: 1045

                Last_IO_Error: error connecting to master 'vrep@mysqlmaster:3306' - retry-time: 60  retries: 86400

               Last_SQL_Errno: 0

               Last_SQL_Error:

1 row in set (0.00 sec)


################################after slave connects:##########################################

mysql>>; show slave status\G

*************************** 1. row ***************************

               Slave_IO_State: Waiting for master to send event

                  Master_Host: mysqlmaster

                  Master_User: vrep

                  Master_Port: 3306

                Connect_Retry: 60

              Master_Log_File: mysql-bin.000004

          Read_Master_Log_Pos: 988

               Relay_Log_File: mysqld-relay-bin.000002

                Relay_Log_Pos: 251

        Relay_Master_Log_File: mysql-bin.000004

             Slave_IO_Running: Yes

            Slave_SQL_Running: Yes

              Replicate_Do_DB:

          Replicate_Ignore_DB:

           Replicate_Do_Table:

       Replicate_Ignore_Table:

      Replicate_Wild_Do_Table:

  Replicate_Wild_Ignore_Table:

                   Last_Errno: 0

                  Last_Error:

                 Skip_Counter: 0

          Exec_Master_Log_Pos: 988

             Relay_Log_Space: 407

              Until_Condition: None

               Until_Log_File:

                Until_Log_Pos: 0

           Master_SSL_Allowed: No

           Master_SSL_CA_File:

          Master_SSL_CA_Path:
              Master_SSL_Cert:

            Master_SSL_Cipher:

               Master_SSL_Key:

        Seconds_Behind_Master: 0

Master_SSL_Verify_Server_Cert: No

                Last_IO_Errno: 0

                Last_IO_Error:

               Last_SQL_Errno: 0

               Last_SQL_Error:

1 row in set (0.00 sec)




################################after slave connects:##########################################


mysql>>; show processlist;

+----+------+------------------+------------+-------------+------+----------------------------------------------------------------+------------------+
| Id | User | Host             | db         | Command     | Time | State                                                          | Info             |
+----+------+------------------+------------+-------------+------+----------------------------------------------------------------+------------------+
|  2 | root | localhost        | ops_backup | Query       |    0 | NULL                                                           | show processlist |
| 34 | vrep | mysqlslave:35958 | NULL       | Binlog Dump |  282 | Has sent all binlog to slave; waiting for binlog to be updated | NULL             |
+----+------+------------------+------------+-------------+------+----------------------------------------------------------------+------------------+
2 rows in set (0.00 sec)






ssh tunnel,make sure ssh keys are set
ssh -N -f -L 5007:localhost:3306 root2@209.207.232.136



Errors:
Slave I/O: Master command COM_REGISTER_SLAVE failed: failed registering on master, reconnecting to try again


soln:
[ERROR] Slave I/O: error connecting to master, Error code 1045

The MySQL troubleshooting page gives a few hints, talking about credentials for the replication user. However, attempting a connection via the command line client works, so the password is OK and the slave can reach the master. As it turns out, for replication purposes, the maximum password length is 32 characters.

max is 32
min is 8



Errors
Can't execute the query because you have a conflicting read lock

Soln
show processlist;


check

on master

mysql>>; update backup_job set filesize=40000 where backup_id=131710;

Query OK, 1 row affected (0.04 sec)
Rows matched: 1  Changed: 1  Warnings: 0


mysql>>; update backup_job set filesize=44800 where backup_id=131710;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0


mysql>>; update backup_job set filesize=44900 where backup_id=131710;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0



on slave

mysql>>; select * from backup_job where backup_id='131710';
+-----------+---------------------+---------------------+----------+--------+------+----------+------------------------------------+----------+-------+
| backup_id | job_start_dt_tm     | job_end_dt_tm       | pod_name | status | type | dateb    | filename                           | filesize | vault |
+-----------+---------------------+---------------------+----------+--------+------+----------+------------------------------------+----------+-------+
|    131710 | 2014-11-07 07:15:15 | 2014-11-07 07:15:17 | drvv12   | Error  | inc  | 20141107 | inc-vault-3971-201411070715.tar.gz | 40000    | 3971  |
+-----------+---------------------+---------------------+----------+--------+------+----------+------------------------------------+----------+-------+

1 row in set (0.00 sec)


mysql>>; select * from backup_job where backup_id='131710';
+-----------+---------------------+---------------------+----------+--------+------+----------+------------------------------------+----------+-------+
| backup_id | job_start_dt_tm     | job_end_dt_tm       | pod_name | status | type | dateb    | filename                           | filesize | vault |
+-----------+---------------------+---------------------+----------+--------+------+----------+------------------------------------+----------+-------+
|    131710 | 2014-11-07 07:15:15 | 2014-11-07 07:15:17 | drvv12   | Error  | inc  | 20141107 | inc-vault-3971-201411070715.tar.gz | 44800    | 3971  |
+-----------+---------------------+---------------------+----------+--------+------+----------+------------------------------------+----------+-------+
1 row in set (0.00 sec)



mysql>>; select * from backup_job where backup_id='131710';
+-----------+---------------------+---------------------+----------+--------+------+----------+------------------------------------+----------+-------+
| backup_id | job_start_dt_tm     | job_end_dt_tm       | pod_name | status | type | dateb    | filename                           | filesize | vault |
+-----------+---------------------+---------------------+----------+--------+------+----------+------------------------------------+----------+-------+
|    131710 | 2014-11-07 07:15:15 | 2014-11-07 07:15:17 | drvv12   | Error  | inc  | 20141107 | inc-vault-3971-201411070715.tar.gz | 44900    | 3971  |
+-----------+---------------------+---------------------+----------+--------+------+----------+------------------------------------+----------+-------+

1 row in set (0.00 sec)