Skip to main content
Browser Warning

Enhance Your Experience Get a newer, better browser (it's free!)

Ready for an upgrade? A different browser will keep you connected to the full website experience – and protect you from security risks.

Close Search

Principal System Engineering - Production Operations

Plano, Texas

Apply Now

build

Join us to create a new era of connectivity.

"AT&T allows me to work on projects that will be seen by millions of customers."

Megan T. — Sr. Specialist, Software Engineer

"I find it incredibly rewarding to be out and see customers enjoying a product I spent my time perfecting."

Technology

Connect communities with the power of technology.

Innovation is the power to Think Big

We live our values without question or compromise.

A Look at Our Design Team

A day in our UX/UI team.

Revolutionize Business in our Digital Team

Transform how employees and customers connect

Job Overview

Job Description:

Join AT&T and reimagine the communications and technologies that connect the world. Our Consumer Technology experience team is delivering innovative and reliable technology solutions to power differentiated, simplified customer experiences. Bring your bold ideas and fearless risk-taking to redefine connectivity and transform how the world shares stories and experiences that matter. When you step into a career with AT&T, you won’t just imagine the future-you’ll create it.

The Principal System Engineering of Operations Tier 1 is responsible for helping lead a team of people dedicated to proactively ensuring high availability, reliability and resiliency of AT&T's customer & agent facing experiences and shared omnichannel platforms.

Responsibilities

  • Provide 24x7 Tier 1 support for customer & agent facing applications operating across eCommerce, Care, & Retail platforms built on microservices based architecture on prem & in Cloud including SaaS: Salesforce, Salesforce Marketing Cloud, MuleSoft, etc.

  • Management of escalated issues, incidents and outages, triage and driving prompt resolution

  • Provide prompt visibility and status of escalated issues, incidents and outages to leadership, business partners and other key stakeholders.

  • Responsible for Site Reliability Engineering aspects such as developing functional and technical knowledgebase of the application, creation of run books, developing observability of the application in terms of alerts, monitoring and dashboards that enable proactive incident and problem detection, triaging of the incidents and helping Tier 2 conduct blameless post-mortems (after action reviews).

  • Oversee daily T1 operations of premise and hosted applications and experiences, including data centers, compute, storage, data networks, monitoring and NOC.

  • Work with Release Management related to upcoming changes to production to identify risks and mitigate them.

  • Work closely with Product Development & Tier 2 SRE teams to ensure Knowledge Transfer related to changes to the system well in advance of change getting operationalized.

  • Optimize the overall T1 on-call process and incident response workflow, including managing the team’s on-call rotation, alert rules, communication methods and incident response plans.

  • Provide metrics and status reports and review with leadership and stakeholder communities; establish processes surrounding metrics gather, reporting and communication.

  • Staying current on feature development and how it could affect the system’s overall reliability.

  • Assist in developing, publishing and continually updating technology operations and support Standard Operating Procedures and detailed T1 documentation based on industry best practices.

  • Provide technical leadership with great communication skills, with an ability to create and organize self-motivated team.

  • Conduct rigorous due diligence on all plans.

  • Drive team engagement. Motivate individuals and teams beyond current scope of influence.

  • Champion and facilitate breakthrough solutions. Take appropriate, intelligent risks.

  • Create, enable and cultivate a culture of responsibility and accountability.

  • Lead by example and operate with transparency, integrity and respect.

Qualifications:

A suitable candidate for this position must possess the following applicable knowledge, skills and abilities. In addition, be able to demonstrate and provide applicable examples to support his/her competencies.

  • Bachelor's degree in Computer Science or Engineering, or a related field

  • 10+ years of demonstrated leadership experience building cross-organizational consensus

  • 10+ years of demonstrated experience building and managing high-performing teams

  • 10+ years of demonstrated experience with Incident Management, Incident response and managing Tier 1 Production Operations team

  • 10+ Years of supporting large scale eCommerce, Care, & Retail POS platforms & supporting applications in production in a leadership capacity.

  • Solid understanding and experience in Application Performance Monitoring tools like Dynatrace, AppDynamics, Introscope, etc.

  • Hands-on experience with Customer Experience Analytics & Session Based tools like Quantum Metric or Tealeaf

  • Hands-on experience with Synthetic Monitoring tools like Catchpoint

  • Experience working within scaled agile development team.

  • Experience developing and implementing customer journey dashboards to enable proactive monitoring of customer experience availability.

  • Experience designing and managing a world-class technical operations organization including 24x7 support and outage/incident management.

  • Solid knowledge of Operations practices and demonstrated experience increasing Operational capability maturity within an organization.

  • Excellent communication and presentation skills; the ability to present complex technical information in a clear and concise manner.

  • Proficient at analyzing and interpreting large amounts of data with the capacity to synthesize information and translate into effective and actionable insights.

  • Exceptional organization and planning skills, strong analytical abilities, and process-driven orientation

  • Unrelenting sense of customer-focus, urgency and accuracy with an execution mindset

  • Self-starter, creative, enthusiastic, innovative and collaborative outlook

Primary technical skills should include:

  • Java, Spring, WebLogic, AKS, and CI/CD tools, PL/SQL

  • Microservices based architecture using Java, J2EE, Jenkins, Maven, Linux, K8s, on both on-prem and in cloud.

  • Docker, Kubernetes and Microsoft Azure Cloud, Unix

  • Relational & NoSQL databases like Oracle & Cassandra

  • Experience with visualization tools like Kibana and Grafana. EFK stack experience preferred.: (Hands-on experience is must)

  • Creation of Dashboards on Dynatrace, ELK and Grafana. (Hands-on experience is must)

Secondary technical skills (optional, yet highly desirable):

  • Salesforce Development (Apex, Visualforce, Lightning), Salesforce Sales Cloud & Service Cloud, MuleSoft, Dynatrace and ELK (Elastic, Logstash, Kibana) for monitoring and logging

  • Hands on experience supporting Salesforce applications

    • Sales & Service Cloud

    • Experience with Marketing Cloud

    • Experience within high tech, software and/or wireless/telecom industry highly desired

    • Understanding of integration technologies and API Gateway, Mobile and iOS technology stack; Experience with MuleSoft desired

  • Solid technical background with understanding and/or experience in software development, web technologies and customer communications such as email, SMS and push notification.

Our Principal System Engineering, earn between $158,200 - $237,400. Not to mention all the other amazing rewards that working at AT&T offers. Individual starting salary within this range may depend on geography, experience, expertise, and education/training.

Joining our team comes with amazing perks and benefits:

  • Medical/Dental/Vision coverage

  • 401(k) plan

  • Tuition reimbursement program

  • Paid Time Off and Holidays (based on date of hire, at least 23 days of vacation each year and 9 company-designated holidays)

  • Paid Parental Leave

  • Paid Caregiver Leave

  • Additional sick leave beyond what state and local law require may be available but is unprotected

  • Adoption Reimbursement

  • Disability Benefits (short term and long term)

  • Life and Accidental Death Insurance

  • Supplemental benefit programs: critical illness/accident hospital indemnity/group legal

  • Employee Assistance Programs (EAP)

  • Extensive employee wellness programs

  • Employee discounts up to 50% off on eligible AT&T mobility plans and accessories, AT&T internet (and fiber where available) and AT&T phone

A career with us, a global leader in communications and technology, comes with big rewards. As part of our team, you’ll lead transformation surrounded by trailblazing industry leaders like you. You’ll be empowered to go above and beyond – making a difference through company-sponsored initiatives or connecting and networking through one of our many employee groups.

And regardless of where you’re at in your career trajectory, you’ll be rewarded by the impact that comes with making a difference in the lives of millions.

With AT&T, you’ll be a part of something greater, do incredible things and be rewarded with a chance to change the world.

AT&T will consider for employment qualified applicants in a manner consistent with the requirements of federal, State, and local laws.

Ready to close the deal on a career with AT&T?

#ConsumerTechnologyeXperience

Weekly Hours:

40

Time Type:

Regular

Location:

Plano, Texas

Salary Range:

$158,200.00 - $237,400.00

It is the policy of AT&T to provide equal employment opportunity (EEO) to all persons regardless of age, color, national origin, citizenship status, physical or mental disability, race, religion, creed, gender, sex, sexual orientation, gender identity and/or expression, genetic information, marital status, status with regard to public assistance, veteran status, or any other characteristic protected by federal, state or local law. In addition, AT&T will provide reasonable accommodations for qualified individuals with disabilities.



Job ID R-34783 Date posted 09/05/2024
Apply Now

Benefits

Invested in your satisfaction and continued success.

We take care of our own here (hint: that could be you). Our benefits and rewards mean we cover some of your biggest needs with some of the coolest offerings. We already think we’re a pretty great place to work. We’re just trying to rack up some bonus points.

Let’s start with the big one: Your work gets rewarded with competitive compensation and benefits. It really does pay to be on our team.

Compensation

When it comes to priorities, we know family tops the list. For the moments that matter the most, you'll be there for them, and we'll be here for you.

Family Leave

Paid Time Off

Our people have class. Literally. We can help you out on approved education costs with our tuition assistance plan.

Tuition
Assistance

Here’s another reason to breathe easy: You and your family get access to excellent medical, dental and vision insurance options.

Insurance Options

Wanna make your friends really jealous? You’ll get discounted access to the latest and greatest AT&T products and services — plus other awesome items, like tickets to live events.

Discounts

You strike us as an over-achiever (don’t worry, it’s a compliment). Our training and development programs are your ticket to expert status in your job.

Training & Development

When the day comes that you get some much needed R&R (not that you’d ever want to leave #LifeAtATT) you’ll know your future is set with the AT&T Retirement Savings Plan (ARSP).

Savings

Give back to your community and connect with colleagues through social and team-building events, and annual paid time off for volunteer efforts of your choice.

Community & Team Events

Wellness resources and incentives to help you prioritize your health and wellbeing and be your best self inside and outside of work.

Total Wellbeing

The Hiring Process

Step 1

Complete a quick application online and check your status often.

Step 2

Virtual or in-person
Interviews

Dress professionally and ensure good WiFi interviewing virtually.

Step 3

Conditional
Job Offer

After a background check, you're part of the team.

Step 4

Welcome! Onboarding
and Training Begins

Our training and certification programs set you up for success.

Here are similar jobs, or

New Search

Discover more at AT&T

Sign up for job alerts, updates and more.

Interested InSelect a job category from the list of options. Search for a location and select one from the list of suggestions. Finally, click “Add” to create your job alert.

Back to top